Is it safe to drive a car with spilled oil on the exhaust manifold?

Wipe off what you can and the rest will dry as the manifold heats up. There will be a noticeable stink as the spilled oil dries up. It should clear after a good drive. On the other hand, a continuous oil leak dripping on or near the catalytic converter could catch fire.