Why did they call the Civil War as a poor man's fight?

The rich men had the greatest stake in the outcome of the war, they stood to lose everything they had if slavery were ended. A poor man though stood to gain very little if they won. But the law made it possible for a man to buy his way out of military service, essentially by hiring a substitute to fight in his place. Naturally you had to be rich to do that and the person you hired was likely to be poor. Thus the rich wanted the war but the poor were doing most of the fighting, and dying.

Rich men also were equipped (as in most wars) with horses, ammunition, and training which would elevate them from the infantry lines.

To add to the above, during the Civil War you could pay someone to serve for you. The poor man literally go to war to fight for someone who was rich.

The Conscription Act of 1863 basically forced the poor men to fight in the war that benefited rich men. Rich men who valued civil rights wanted to abolish slavery, but the poor had to fight, since they didn't have the $300 to pay out of the draft. The poor were actually against slavery, since freed slaves would mean competition for jobs, but they were the ones having to fight the war.