Where in the Bible does it say a pastor has to be a man?

It doesn't, but some will argue that St. Paul said something to the effect that women should keep quiet in church and if they have any questions, ask their husbands at home. St. Paul didn't appear to have much use for women.

1 Timothy 2:11, 12 - Let a woman learn in silence with all submission. And I do not permit a woman to teach or to have authority over a man, but to be in silence.

1 Timothy 3:2 - A bishop [pastor] then must be blameless, the husband of one wife, temperate, sober-minded, of good behavior, hospitable, able to teach;

The New Testament makes a distinction between a preacher and a pastor that many modern Christians fail to discern. 1 Timothy 2:12 regards both, while 1 Timothy 3:2specifically addresses the latter. According to these (and other) scriptures, a woman may not hold either position.

This is not Paul's personal opinion. It should be remembered that he wrote by inspiration of the Holy Spirit, and his instructions are according to God's will.

Acts 5:29 - But Peter and the other apostles answered and said: "We ought to obey God rather than men."

[Quotes from NKJV]