Is a man a pervert if he wears women's underwear?

No. He may be confused about his sexual identity or just like the feel of it. It may seem odd to you.