Is it worth having a wasp nest destroyed at the end of summer if the wasps have finished with the nest?

Wasps do not reuse old nests, so if you know the wasps have gone there is no risk and you can simply remove the nest.