Is it rude to turn down a baptism invitation?

It is not rude. If you don't feel comfortable attending a baptism invitation or do not feel the need to attend for any other personal/belief reasons, you do not have to.