Why does arthur feel that he and hester are not the worst sinners in the world?

Arthur Dimmesdale is a strong pillar of the community and a very devoted Puritan. What could he do that is worse than young Hester Prynne's appalling act of adultery