How did brick making start?

Bricks that you see in houses for the most part are made with sand, clay, iron oxide, lime and magnesia. However, you can also make softer bricks out of just plain mud that you have shaped and let dry.