What were some of the conjectures made by the parishioners as to the origin of the minister's scarlet letter?

They assumed that he (Dimmesdale / the minister ) was indeed the man who had an affair with Hester Prynne and fathered her only child Pearl.