What did cavemen use as a toilet?

By and large, the great outdoors. Eventually, they learned to dig pit or trench latrines to bury the waste. Both of which are far more sanitary and safe than the Medieval practice of dumping it in the streets.