Why are rectangular coils used in power transformers?

Transformers are used to step up and down voltages. The amount of this step is directly proportional to the turns ratio from one winding to the other. Using circular coils means a minimum amount of copper can be used, since the number of coils is relatively fixed by the voltage levels involved.

Another reason is corners are not a good idea when dealing with really high voltages - they can result in eddy currents, strange flux paths, corona, and localized heating to name a few problems.