Is Intel DG965RY DDR3 compatible

No it isnt. You would either need to get a new motherboard or get DDR2 RAM modules to make it work.

Heres the official intel page regarding your motherboard : http://ark.intel.com/products/22637/Intel-Desktop-Board-DG965RY