Does it matter what time you go to bed as long as I am consistent?

Time is relative. 4am for you is 10pm for someone else. There is no harm in going to bed at any time of the day, so long as you keep a few things in mind:

-You need deep, uninterrupted sleep to stay healthy.

-You should get between 6 and 8 hours every night if you can.

-Vitamin D processing in the human body has a large connection to the amount of sunlight you get. If you find you are not being exposed to the sun because you are nocturnal; then it is a good idea to take a vitamin D supplement, or drink milk. A lack of vitamin D can cause mild depression among other things.