How does the mass of an object accelerate during free fall?

When an object is falling in a gravity field, its mass does not affect its acceleration. If under the influence of friction then it takes more energy to accelerate it the larger mass it has, here on Earth. The larger the mass the more potential it has to do work if it is above the surface of the earth than a smaller mass. But in turn, to get it to the higher point more work must be done for a larger mass than a smaller mass.