Why is Ebenezer Scrooge down cast at Fezziwig's party scene?

He remembers how Fezziwig treated him and how well he looked after all of those who had little or nothing and then Scrooge thought of Bob and his treatment of his loyal employee