What do you call the baby of your brother?

The baby of your brother is called your nephew if it's a boy and your niece if it's a girl. These terms denote the relationship between you and your brother's child.