What would you call your sister's son?

The English language does not have special terms for a relationship involving an older, or a younger, brother or sister.

Your brother's son is your nephew and you are his uncle, if you are a man, or his aunt, if you are a woman.