What does an executor of estate do when a will wasn't made and there is a lot of unknown debt Is it mandatory to post who the executor is in the newspaper so these debts can be paid?

That is one of the primary purposes of creating an estate. The executor has to contact all known debtors and advertise for unknown debtors. There has to be a method for these debtors to contact the executor to place their claims.