What is the real difference between a pepper mill and a salt grinder?

Very little. Both take big ones and grind 'em down into smaller ones. In many cases, the internal workings are identical. If I were to be pressed to it, I would say that the main difference is that a salt grinder must be made of stainless steel, where a pepper grinder can be made of mild steel.