Do you capitalize thee when referring to god?

Yes, you do.

That is right; the normal spelling convention is to capitalize all pronouns that refer to God, which includes the pronoun thee, if you are addressing God. Nearer, my God, to Thee.