What challenge did early farmers not face as a result of the change from a nomadic lifestyle to an agricultural lifestyle?

Housing could become more solid and more of a protection from the elements. Food availability would become more predictable, increasing survival of infants and children.