Are all the details in the infancy narratives historically true?

Well, they can't be, can they?

In Matthew's Gospel, the home town of Joseph and Mary was Bethlehem. After the birth of Jesus they fled from Bethlehem to Egypt with Jesus, for fear of King Herod, who ordered the slaughter of all the boys under two years old. After the death of Herod, they began to return home but, being warned in a dream, turned aside and travelled Galilee where they settled in a town called Nazareth "that it might be fulfilled which was spoken by the prophets, He shall be called a Nazarene (Matthew 2:23)". At least in respect to the visit of the magi, John Shelby Spong (Born of a Woman: A Bishop Rethinks the Birth of Jesus) says that Matthew was clearly writing Christian midrash.
Compare Luke's Gospel, where the home town of Joseph and Mary was Nazareth. Augustus Caesar reportedly ordered a census of the whole empire so that everyone could be taxed - although historians know of no empire-wide census during the time of Augustus. This required Joseph and Mary to travel to Bethlehem, where Jesus was born in a stable. Shortly after the birth of Jesus, he was taken to the Jerusalem Temple, where Jesus seems to have attracted a great deal of attention, and the family then returned peacefully to Nazareth, without attracting any interest from Herod. Moreover, Luke says that Jesus was born during the reign of King Herod, who died in 4 BCE, but also during the time Qurinius was governor of Syria, a position he held after 6 CE. Raymond E. Brown (An Introduction to the New Testament) says that the best explanation is that, although Luke likes to set his Christian drama in the context of well-known events from antiquity, sometimes he does so inaccurately.