Why are z-scores used to check for outliers?

If a variable is Normally distributed then the z-score describes how far from the mean/median a particular observation is. For example, a z score of 1.96 implies that fewer than 0.025% of the observations will be at least that extreme.