What is a group of four books called?

One Word is quartet.

Some others are tetrad and foursome.

If they are four people, a jocular term is "gang of four".