Why is build up of lactic acid a problem for athletes?

It's because lactic acid is a mild toxin. Reduced oxygen in the bloodstream causes lactic acid to build up in the body's tissues. This causes muscle cramps which forces you to rest, to give your body time to recover.