Why did jem need to return to get his pants in to kill a mockingbird?

If he hadn't returned to get his pants, they would have found out that Scout, Dill and him were at the Radleys and were the kids that Mr.Radley was about to shoot