What do the townspeople think The Scarlet Letter means?

I'm pretty sure most of them had a clue what Hester was accused of , as no decent young woman should have been in some unrelated man's house anyway .