Is clay cliffs made of clay?

Yes, that is why they are called clay cliffs.

When it dries, clay is firm, and forms a soft rock, however it is easily eroded, so clay cliffs are often unstable.