Why did hydrogen and helium leave earth's atmosphere?

There's a more scientific way to explain it but heres a simple one; If you have a balloon filled with helium where does it want to go? Up because its lighter than air so earths gravity is having a lesser effect on it. SO if its not in the confines of the balloon it will keep going until it reaches the outer edges of the atmosphere, at this point it will slowly 'leak' off into space and it will do so first as its lighter than most other gases floating around in the upper atmosphere