What does the name graybacks refer to as it was used by troops from both sides in the civil war?

In war, enemy soldiers are given nicknames. Graybacks was the Union nickname for Confederate soldiers because of their gray uniforms, just as British soldiers were called redcoats because of their red uniforms.