Not only was the Union army recruiting blacks to join their forces, they were also impressing slaves from both Unionist and Confederate owners. Often making those that didn't want to enlist serve anyway.
A short article appeared in the Frankfort Tri-Weekly Commonwealth on March 4, 1864 under the headline, "Let Justice be Done!" explained the unfair treatment Unionist slaveholders felt they were receiving at the hands of the federal government.
"General [Stephen G.] Burbridge has issued an order extending to the impressment of negroes to other counties than those included in the order heretofore. Now, General, we think it decidedly wrong that the order includes the negroes of loyal as well as disloyal persons. Let the negroes of loyal persons remain with their owners, and impress only the negroes of the rebels and their sympathizers. The necessity of impressing negroes and other property alone rises from the rebellion, and the aiders and abettors of the rebellion are the ones from which the impressments should be made. Let justice be done the Union people, by letting their property alone. Let justice be done the rebels and those who give them aid and comfort, by taking their negroes and property for Government use."
This perceived injustice and hard-handedness toward the loyal people of Kentucky was one of several reasons that the commonwealth came to identify itself more with the Confederate cause after the shooting stopped. Like the people of the Confederate states that seceded over the Union's threats to slavery, those in the slaveholding border states too associated liberty with property, only they felt their best chances to remain slave owners was to stay in the Union, not leave it. The Emancipation Proclamation, although it technically did not effect them, started the slippery slope that ended the institution with the 13th amendment to the Constitution. Enlisting and impressing former slaves were just more nails in the institution's coffin.