Are all babies born as Muslims?

No they are not. There are many religions, so you can certainly assume that a baby is, or will be, the religion that their family practices.