Why does the community shun Dolphus Raymond in To Kill a Mockingbird?

He is white, but lives with blacks. They somewhat accept it because they all believe that he's constanly drunk and therefore does not understand his wrongdoing.