If Neanderthals had larger brains why were they unable to compete within modern humans?

The larger brains had the result of making Neanderthals more aggressive and bullying but less cooperative. The smaller brained, cooperative, species killed off the aggressive bullies and their children.