What is the difference between factory and plant?

It might just be semantics. The Words can pretty much be interchanged (and often are), but the difference is probably that a factory may have several plants within its manufacturing complex. A car factory might have an engine plant and an assembly plant as well as a bunch more other plants, each of which puts together a subassembly of the vehicle. It is not surprising that if the definition of each work is looked up, the other Word is often applied in that definition of that Word.