Were there only three wise men?

The actual Word used in Matthew is magoi or, in Latin, magi (sing: magus), which really refers to priests of the Zoroastrian god, Ahura Mazda ("Wise Lord").

Matthew was portraying the priests of this great religion as regarding the birth of a Jewish Messiah as a momentous event. By having them wish to worship him, Matthew was probably demonstrating that Jesus had dominion over even the Zoroastrians. In later centuries, the Zoroastrian connection became less important and the term "wise men" tends to be used in English translations. In tradition and nativity plays the priests are sometimes portrayed as kings.

John Shelby Spong (A Bishop Rethinks the Birth of Jesus) says that among people he knows in New Testament circles, the universal assumption is that the magi were not actual people. Matthew was clearly writing Christian midrash.