What is the correct term for you to call your brother-in-law's grandson?

Your brother-in-law is your spouse's brother. His grandson s your spouce's great nephew. (Some people say grand nephew). That makes him your great nephew, too.

Your brother-in-law could also be the husband of your sister. Then your brother-in-law's grandson is your sister's grandson and your great nephew.