Some men and women enjoy a change in those they date or may cheat with and yes, at the time they find that person physically attractive, but most men aren't sizing up the mistress with the wife. Men generally want to change their environment and to be looked up too; feel young again or feel free to do as they please, but eventually these types of men grow weary quickly of even their mistress and once he has cheated he will cheat on his mistress too or, perhaps be lucky enough to realize that the grass in not always greener on the other side and go back to his original partner if she will have him.
