When the civil war began in 1861 how many freemen were living in the US?

Most were living in the south, where there were half a million free blacks, and three and a half million slaves. There was no large black population in the northern states.