What side did the border states support?

They supported the North - with some reluctance, as there was a lot of pro-Southern sentiment in those states.

In Kentucky there was briefly a Confederate government installed by General Braxton Bragg, which collapsed once Bragg's army had retreated.