Why do people associate certain music with sadness?

People will associate anything with anything, not just music. The mind just connects things.

Music is highly emotional. In the West, music is divided in major and minor keys. Musical expression of the minor third is the one which seems to connect with the human emotion of sadness the most strongly. Major key music conveys a happier, positive emotion.