Why is sand used for construction of buildings?

As sand is a course granular substances binding agents such as cement and lime can grip to the sand and form stronger bonds - if the mix was all powdery like a cement it would be more line a slurry or even a doh like consistency which would not hold very well under pressure and in weathering. When buildings were first created using mud (wattle and daub) course materials were put in the mix like animal hair and even manure to brake up the slurry like consistency and add strength to the mixure