Can a man have a female body and not have a sex change?

Yes, some people are born with a birth defect that can cause a man or a women to both have female and male hormones.

such as a man growing breasts, or a women with a beard.