Why did the community in the giver give up the ability the see color?

They have given all their memories to the receiver so that they would not experience pain. In those memories, were the colors before they decided on Sameness. Therefore, only the receiver/giver only see colors for they are to hold onto all the memories from generations back and from their present time.