Was there ever a small size nickel minted?

The US nickel has been essentially the same size since its introduction in 1866, about 21 mm. There were minuscule adjustments during the first few years of issue but that's all.

The US also issued a different 5-cent coin called a half-dime that was much smaller, because it was made of silver and copper instead of nickel and copper. This coin was discontinued back in 1873.

If your coin looks like a modern nickel but is much smaller it's almost certainly a replica made for a set of novelty items rather than a real coin. These are quite common but can cause confusion if they're broken out of the sets they came in.