Can you refuse to be a trustee?

Yes. When a person is named as a trustee there is generally an acceptance executed by the trustee and filed with the trust. You only need to make if known you do not wish to serve and the trustor can name a different trustee.