Why is the pitcher considered the most important defense player on the team?

Because where and how the pitcher throws the ball has a lot to do with where and how hard the batter hits it.

If the pitcher is throwing the ball on the corners of the plate, moving the ball up and down in the strike zone, and changing speeds generally the batter will not hit as hard as the pitcher who throws the ball, no matter how hard, over the middle of the plate.

The defense will usually set its positions based on how a pitcher is going to pitch to a batter. If the defense plays a batter to the opposite field and the pitcher throws a pitch on the outside corner at the knees, the batter is usually going to hit that ball to the opposite field, maybe sometimes up the middle. If the defense is playing the batter to the opposite field and the pitcher throws the ball belt high over the inside half of the plate, chances are the batter will hit the ball very hard and pull the ball away from where the defense is setup.