If you change your panel do you have to rewire the whole house?

Not necessarily. It would depend on the condition of the wiring. If it is an older house with cloth covered wires it may be advisable to rewire. If existing wiring is aluminum you might also consider rewiring. Depending on the length of wires that enter the current box, you may need some external junction boxes.