Will a weak battery cause starter to keep turning?

No, just the opposite since it take a lot of power to turn it. Sounds like hot terminal on solenoid getting power independent of key. See if pos batt cable touching it or is jumping across.