What layer of the OSI model does RTP operate?

It is on the Session layer. Layer 5. RTP has no transport mechanism. If you think of SIP communications (set-up and break down of VoIP calls) this is done via RTP packets. RTP is encapsulated within UDP (for transport) and therefore resides at a layer above the transport layer (layer 4).