What was the music that the medieval monks and nuns sang called?

Your question is phrased rather strangely - perhaps you mean "What accompaniment was there to monk's singing?". They obviously sang with their voices, but you must have some other thought in mind.

The answer is there was no musical accompaniment; singing was called Plainsong or Gregorian Chant and was entirely a cappella, without any instruments. There was also no harmony, descant or bass; everyone sang exactly the same note at the same time (hence Plainsong).

The only departure from this rule was that occasionally a piece would be started solo, then the choir would join in; the soloist might have additional parts later in the piece.

The link below takes you to a YouTube version of Dies Irae sung exactly as it was in the 13th century: