Which side in the Civil War had the most slaves?

The South.

The only slaves in the Union were in the four Border States of the Upper South (Kentucky, Missouri, Maryland, Delaware), where there was less pro-slavery sentiment than in the Deep South.

But of course it was the North that received a constant influx of liberated slaves, who were eventually absorbed into the Union armies.