In lord of the flies Why is Ralph afraid no one would come if he blows the conch?

When Jack and his hunters break off of the group, Piggy encourages Ralph to blow the conch and bring the meeting back together. Ralph chooses not to, saying that if the hunters choose not to come back, the whole order of their small society will be destroyed and he'll have no control.