Do all computers have hard drives?

No.

Some computers use SSD's which are similar to memory but which are non-volatile (ie. it retains information even after power has been cut off). However, they do serve the same purpose as hard drives.

In addition, very old computers (from around 20 years ago had the option of being booted and run from floppy disks, although this usually led to limited functionality.