Is crumbly an adverb

No, it is not. It is the past tense and past participle of the verb to crumble (to fall or break apart, into pieces). It can be a verb form, a participial, or an adjective (e.g. crumbled cheese).