Is the hyena in the dog family?

No.

Hyena are the animals of the family Hyaenidae of suborder feliforms of the Carnivora. It is the fourth smallest biological family in the Carnivora (consisting of four species), and one of the smallest in the mammalia.