What would my nephew's children be called to me?

The traditional terms were "great-nephew" and "great-niece" but grandnephew and grandniece are identically defined: the children of a niece or nephew, i.e. grandchildren of a brother or sister. The terms may have first been used circa 1804.