Did Charles drew go through hard times?

No, but as an African American, he was treated differently. He grew up in Washington D.C.'s Foggy Bottom neighborhood in a middle class family.