Why scientist use control groups?

Scientists use control groups so that they have something to compare or contrast their independent variable to. Without a control group, they wouldn't have anything to prove if their experiment had any important change. For example, a scientist is trying to prove that the bugs in his garden is affecting his flowers. His control group would be a set-up without the bugs. He would have another set-up WITH bugs. After some time, he would compare the two. If he didn't have a control group, how would he know that the bugs affected his flowers at all? Maybe the flowers were changing due to another factor, but how would you know?