What reason does Hawthorne give for losing his job in the custom house?

Hawthorne cites political changes as the reason for losing his job in the custom house. He was a Democrat and when the Whigs came into power, they replaced him with one of their own appointees.