When do you remove the flag when on the green?

At any stage, if you are on the green, or even if you are off the green. However, be careful, if your opponents ball is in motion, and you remove the pin without their permission you receive a two shot penalty. Also if you are on the green and you hole a shot with the pin in, or hit the pin, you will receive a two shot penalty.