Why doesnt The Giver leave?

Because he needs to help the community when the memories come out and when the Elders find out they will probably release him and he wants to be with his daughter Rosemary