What is a brother son called?

The son of either your sister or your brother is your nephew. The English language does not make a distinction in that.