Can you use lavender fragrance oil for skin?

You can, if you're referring to the essential oil (usually found in a 10ml bottle), but it's not highly recommended. I would recommend even adding two or three drops to 1ml of a carrier oil just to be on the safe side. If you're referring to a lavender oil from Glade or some major brand like that, the simple answer is: absolutely not.

That said, lavender and tea tree are the ONLY two essential oils that can be applied neat (or in other Words, not diluted) to the skin. This does change when it comes to elderly or infants however due to more sensitive skin, but generally has no adverse side effects in healthy adults.