Why were some Southerners against tariffs?

Because the southerners had built few factories and didn't benefit from the tariff. Southerners bought many British goods and the tariff drove up the price. The southerners complained that the tariff made northern manufacturers rich at the expense of the South.