Will a starter smoke before it goes out?

Normally starters do not smoke before they fail but if it does smoke it will soon go out either from overheating or if it has oil inside it causing it to smoke. Oil leaking from the valve cover could get inside the starter in short it out. Oil inside the starter could cause it to smoke and will soon cause it to fail. Be sure that the heat shield is installed over the starter or it will soon over heat and fail. Properly installing the heat shield is a little difficult so many times unprofessional mechanics will short cut and leave the heat shield off and cause a very very early failure.