Were there giant humans back then?

The occasional larger-than-average human has probably popped up every now and then throughout history. But there's never been a whole society of giants.