Was Shem at the tower of Babel?

It would appear from the Book of Jasher that Shem was not at the tower of Babel. He lived with Noah for the rest of Noah's life away from the pagan idol worship of the day. Abraham lived with them a number of years to learn the one true and living God and the correct ways to worship Him. Book of Jasher Ch.9