Function of earth fault relay

An "earth fault relay" is a bit ambiguous. A relay used in the power system to detect neutral or ground faults measure the vector difference of the three phase power, or measure the neutral current directly. If current is above a set trip point, the relay will operate.

If you are referring to GFCI's, they effectively measure the current flowing in and the current flowing out on the two "hot" wires, and if these do not cancel each other out, then the GFCI will trip. This is because if current in does not equal current out, then some current must be flowing out a different way (to ground!).