Is downloading only the audio of a Youtube video legal?

It depends if the audio of the video is protected by copyright. For instance, if you are talking about using YouTube to download music, (which is what I'm assuming you have in mind since one of your tags is 'iPod,') then no, it is not legal. Let's say, however, you want to download the audio of a news report or something. That would be perfectly legal.