What does it mean when you and your friend are drifting apart?

It means that you are taking two different paths and are not as close as you once were - your both changing directions and interests.