Do you have to signal when there are no cars around?

Yes, you should.

You do not only signal to let cars know where you are going, but you also signal to let pedestrians know your intended direction, to stop them walking in front of you.

You may have also missed abicycleor motorcycle in your mirror check, since they can be difficult to spot in some conditions. You should signal when necessary for your own and others safety.