Why did prisoners of war suffer so bad in both the Northern and southern camps?

Neither side made provisions for large numbers of prisoners. Guarding prisoners was a secondary consideration to winning the war. At the beginning of the war, there was a system of prisoner exchange and parole which eliminated the need for prison camps, but the system broke down when the South refused to parole African American soldiers. Once this happened, northern generals cancelled the parole program altogether.