Why were abolitionists so hated by the south?

Southerners hated abolitionists because they would speak out against slavery. If there was no slaves than no one would work the fields for the rich men.