What do you call your sisters son or daughter?

Short answer: They would be your niece and nephew.

Long answer:

Your sister-in-law is either your sister-in-law because she is married to your brother (or, in same-sex marriages, your sister) or because you are married to her brother (or sister in same-sex marriages). If it is the former and she married your sibling, then her children would be your niece and nephew by blood--you share the same gene pool--happy swimming). Or, if it is the latter, and you married her sibling, then they are your niece and nephew by marriage and you do not share the same gene pool--unless there is another familial connection. Either way, you still have to buy them Christmas and birthday presents!