Should you close a flue on a wood burning fireplace when not lit?

If you do not close it then you basically have a hole in your roof that is sucking out all the warm air. This will lead to even higher heating and cooling costs.