How is the Curse of Ham associated with black people?

Genesis 9:20-27 says that Noah's son Ham "saw the nakedness of his father" and God responded "Cursed be Cannan" (Ham's son) and went on to bless Shem (Noah's other son, for protecting his father's nakedness) and state that Cannan's descendants would be servants to Shem's. This story appears to be told as a justification for the much later Israelite invasion and subjugation of the Land of Cannan. Later, when race-based slavery emerged, racists re-interpreted this story to justify race-based slavery by concluding that Africans were descended from Ham and therefore cursed by God to be eternal servitude.