What do you do if your brother is gay?

There is no way. Except the stereotypical signs but most if not all of them aren't reliable. The only way is to ask him and if he trusts you enough he'll come out to you if he is gay. If not you have to take his Word for it!

It would be better if you were to just ask him about his sexuality. Though, he may be uncomfortable talking about it.