Who did Scrooge's nephew say suffered by Scrooge's ''ill whims''?

Fred, the nephew, said that Scrooge himself was the only person who really suffered from his misogynistic habits.