Why did The men from Old Sarum who come the jail leave?

The men from Old Sarum who came to the jail left because they realized they had the wrong person. They were looking for someone else and had mistaken the identity of the individual in the jail.