Why did Mr. Dimmesdale rush from the room to end the conversation he had with Roger Chillingworth in the scarlet letter?

In The Scarlet Letter by Nathaniel Hawthorne, when Chillingworth prods Dimmesdale about his spiritual condition, he becomes agitated and rushes from the room because he feels that such matters are the concern of God and not man. ChaCha.