Are all living things made up of millions of cells?

No, there are living things, such as bacteria, which have only one cell, each cell being an individual.

It's generally accepted though that the smallest and most basic unit of a living organism would have to be one cell, which would exclude viruses and the like. There are broader definitions that include them, however.

(Viruses don't have any cells, they're basically a lipid-proteic capsule and RNA or DNA).