Why do Macbeth's followers start to desert him?

"That is a step on which I must fall down or else o'erleap, for in my way it lies." Macbeth feels that he is the right person to succeed Duncan, in part because the witches have suggested it to him, but in part because he is in line for the throne, and is a proven warrior and capable leader, whereas Malcolm is neither. The system in Scotland at the time was not one of simple primogeniture, or otherwise Macbeth would not have been dismayed at Duncan's appointment of his feckless son as Prince of Cumberland, thus naming him as his heir. Had the inheritance of the throne been clear and obvious, there would have been no question of who would succeed Duncan from the very start.