What makes a war illegal?

When the combatants are NOT recognized by the people they are fighting. Normally only a recognized government which consists of courts, police, a capital city, a President/King/Emperor/Chairman/etc., is a recognized government. Then the rules of war can apply. If it's simply a group of people creating violence with guns, hand-grenades, etc. then they are criminals or terrorists or insurgents...all of which can be considered illegal. Those particular people would not warrant protection under the rules of war...they would warrant protection under the criminal justice system.