What is the significance of Jem returning to the Radley place to get his pants?

They had been stitched back up (poorly), folded, and left on the fence. Line from the book says something like "as if someone was expecting me to come back to get them".