Which group do most scientists consider to be direct ancestors of modern humans?

I think it depends on how far back we're discussing. Homo Heidelbergensis seems to be regarded as a direct ancestor.