Limitations of using least square methods in analysis?

One of the limitations of using least squares methods in analysis is that outliers, which are significantly bad observations, can skew the results because they have more impact. This impact is because the square of a number grows large faster than the number. It is better to reject the outliers using some other method prior to using least squares on the remaining data. Of course, this must be substantiated because rejecting data otherwise is bad practice.