Can a Mexican citizen who has a Matricula but not a green card travel from California to Hawaii?

No. NTSA guidelines require a passport (foreign or US), US passport card, green card or US drivers license or non-drivers license.