What would happen if a voltmeter were substituted for an ammeter?

Smoke. Since a voltmeter is in parallel with the load it is right across the source voltage. Putting the amp meter across the line with its low resistance it will act like a fuse, hence the smoke. Newer solid state testers are usually smarter that the operators. They have built in circuitry which sense the wrong settings you are using and shut the tester off with a "beep" to let you know that you are doing something wrong.