Do you need to change both of them to get the light to go out?

Yes, you typically need to change both bulbs if they are part of a series circuit; if one bulb is burned out, the circuit is incomplete, and the light will not go out. In a parallel circuit, however, changing one bulb may suffice, as the other can still complete the circuit. Always check the configuration of the light fixtures to determine the correct approach.