Is your brothers grandson your grand nephew?

No, your brother's grandson is not your grandnephew; he is your nephew. A grandnephew is the son of your niece or nephew. Therefore, your brother's grandson is simply your nephew's son.