Is didn't an adverb

Didn't = "did not". The "not" part is an adverb.

The "did" part is an auxiliary verb, used to reinforce whatever verb it helps out.