Can you lose your medical license due to adultery?

Actually, if the grounds count where the doctor has breeched the code of ethics of Medical doctors then Yes, it is a possibility and there have been cases of it. It may also depend on the rules and laws of a certain country.