Why did slaveholders claim that the Wilmette proviso was unconstitutional?

Slaveholders claimed the Wilmot Proviso was unconstitutional because they argued that it violated the Fifth Amendment rights of slaveholders by depriving them of their property (slaves) without due process of law. They believed that Congress did not have the authority to ban slavery in the territories.