Is it true that Any piece of data that is stored in a computer's memory must be stored as a binary number?

Yes.

A computer's memory is nothing but a bunch of switches... "On" or "Off"

So everything must be represented in a series of 1's and 0's, representing the "on" and "off" of the bits.

The bits are in groups of 8, called a Byte.

So, for example, if you were to save the letter "T", it would be saved as:

01010100