Which side of the Civil War had more men?

The Union - very much so.

It was probably their biggest single advantage - a bigger population from which to recruit armies.