Who fought the confederates?

What remained of the United States Army, after some of the best officers and soldiers had deserted it for the Confederates.

Plus all the new Northern volunteers.