How are Mr Dolphus Raymond and his children a reminder of the racism of both blacks and whites?

Even though Mr. Raymond is one of their own, the white citizens of the town look down on him because he chooses to associate with blacks. It's not just that he's around them; I think they could deal with that. But he chooses to ignore the social norm, and the town can't deal with anything breaking the unspoken rules they've set up. He chooses to be different, and they simply can't handle it.