If a group of women is called a harem what is a group of men called?

The Word Harem is Hebrew for forbidden, so the Word is not specifying man or woman. It was however used to describe the quarters of women as being Harem or forbidden. The men who had women as slaves, servants, and wives owned them as property so they were also considered Harem property.

If you are talking about the collective noun for men and women the Word for a group of women is a "gaggle" of women. For men it is a "band" of men.