What happened if the feudal relationship with a lord and vassal was broken off?

In ancient times, the feudal relationship assisted the serfs in survival. Even though it was difficult and repressive, breaking off the relationship could cause starvation.