How do you unlock the right gate in Fairfax Gardens?

You can't. I can't remember if you get it opened for Lady Gray or during the Fairfax Castle Quest. I remember that you press down on the floor button inside the gate and can walk out but then the gate closes behind you.