Definition of your side of the fence?

Well many fences are made with panels overlapping laths which have a rough and a smooth side (the rough side is the one where the supporting structure of the panels is visible).

The convention is that if, on the deeds of the land, maintenance of the fence is your responsibility (you own it) then you erect the fence such that the neighbour sees the smooth side (out of courtesy).

The fence therefore physically divides a piece of land between the ownership of two people and "your side of the fence' is the side with the land which you own. Further depending on who owns the fence your side of the fence could be rough or smooth.