How do scientist know that planets are much closer to earth than stars are?

By observing their movements. It is actually quite simple: If you watch a planet for two consecutive nights, you can already detect movement among the background stars. You can do this with the naked eye - no special instruments are required. Stars, on the other hand, seem to be unmovable, year after year. It may take decades or centuries to detect a change for some of the nearer stars.