Why did joe frazier not serve in the army?

He didn't get drafted because he had a wife and children but he said if he was he would go to the Vietnam War.