How many pounds do you have to lose to lose 4 sizes?

It depends on several variables. For example, muscle weighs more than fat. Therefore, the muscle to fat ratio can make a difference.

The vanity sizing scale, also known as size inflation, matters (this is the phenomenon of ready-to-wear clothing of the same nominal size becoming larger over time) as does the material used for the garment, the cut of the garment within one size varying due to a different pattern, and the machines or humans doing the cutting and construction.

Overall, perhaps 14 pounds will bring the average person down a size. As already stated, it will vary.