Why were Ham's descendants punished when they tried to build the Tower of Babel?

Many people miss the point of the Babel story. The belief that the builders were trying to get to Heaven is a bit simplistic, and totally incorrect.

God punished the builders because of their arrogance, their belief that they were equal to God.