What happened to Adam Smith at the age of four?

According the Scottish biographer and journalist John Rae, Adam Smith was abducted by gypsies at the age of four. Smith was eventually released when others went to rescue him.