Why was Odysseus displeased with Penelope?

Odysseus dodn't necessarily "leave" Penelope. He was advised by Athena to fight for Ithaca in the battle against Troy. Odysseus was reluctant to do so, for his wife had just given birth to a baby boy, Telemacus. He told Penelope that if he had not returned by the time that Telemacus had a beard, he was dead and that she would have to remarry to keep the kingdom.