Why are carbon dioxide and carbon particles more likely to be formed in engines than if the fuel was burnt in the open air?

Your question assumes that it is more likely for this to happen in engines. It is NOT more likely.

Burning a fuel anywhere produces carbon dioxide. The formation of the carbon particles depends on the amount of oxygen available and the tepmerature of combustion.