Why does a partially inflated balloon expand as it rises?

The pressure in the balloon stays the same, but the pressure outside drops as the altitude increases. And as the outside pressure drops, the balloon expands.