Why don't the townspeople like the Radley's To kill a mockingbird?

Down south, everyone or close to everyone is racist. Even if a white person was wrong, they would still blame the black man or woman. So they were angry because atticus went against it because he isn't prejudice