Why does society think it's so wrong for men to wear women's clothes and for women to wear men's clothes?

If you think about it, most people in society don't care if women wear guy clothes. It's the men that wear women's clothes that bother them. It's most likely because the majority of our modern society is unfortunately homophobic; and they figure if a man is willing to wear women's clothing he is obviously a homosexual.

It also has to do with the "Male Privilage" most western societies continue to hold onto. When a girl embraces her Tommy boy side, it is not frowned upon as she is attempting to improve her status in society. Where is a male member of society were to shed his male privilage others in the society feel he is being "crazy."