Is it illegal to download YouTube videos?

Yes. Section 5b of the YouTube terms of use states "You shall not download any Content unless you see a “download” or similar link displayed by YouTube on the Service for that Content."

However, it has been upheld that content that is obtained from a legal source and used for personal, non-commercial use is legal. This was decided in the Betamax case, which had to do with the rights of home recordists taping TV programs for personal use. Such copying qualifies as fair use under U.S. copyright law. The same principles would apply to content taken from the internet, provided the content is legal to begin with.

You'll get differing opinions on this. If you listen to the MPAA, the RIAA, and probably the AAA, they'll tell you that it's illegal to copy anything, even if you're a cloistered monk working on an illuminated manuscript. However, this is their opinion, which thus far has been unsupported by the case law. If, however, you know that the content you're downloading is pirated from someone who would object to its use - e.g., a bootleg copy of a film currently being shown in theaters, you could be complicit in the crime, even though it's unlikely you'd ever be prosecuted for it.