Why is nitrogen used in Nitrox?

Nitrox is just oxygen enriched air.

If you used pure oxygen, it would be toxic past a depth of 20 feet / 6 meters.