Is it wrong for straight men to wear womens clothing?

Certainly not. It is only wrong if it feels wrong. You should wear what makes you feel most comfortable.

Asserting that this is wrong would be like claiming it is not okay for me to wear sweaters because only old ladies wear sweaters, or wearing the color grey is indicative of poor intelligence. The two have nothing to do with each other and I should dress in a way that I am happy with and that best suits me.