How can the behavior of two magnets show the presence of a magnetic force?

suspend the magnets from strings in such a way that they only repel each other. (Between two glass plates maybe, so they can't rotate.) measure the angle of the strings.

Knowing the weight of each magnet, the acceleration of gravity, and the cosine of the angle between the strings this should give you enough to calculate the magnitude of the force between the magnets. (Providing you know vector addition of course.)