How do you defend a castle from a catapult?

If I had to defend a castle then I would either build the walls stronger so catapults or other modern day objects Can not get through the Walls of the castle or make the castle higher so the stones or other objects cant get over the Wall's.