Can you fly out of Australia on an Australian passport and enter the US on a Canadian passport?

You absolutely can. You should also exit the US on your Canadian passport if you do so.

Canadian and Bermudian passports are the only passports that have visa-free access to the US and yet are not required to register for ESTA before their departure.