Was the first clue that the Giver himself had depth?

Yes, the first clue that the Giver had depth was when he displayed emotions and memories that were absent from the rest of the community. This hinted at a complexity and individuality that set him apart from others.