Who would live in a motte and bailey castle?

not very often now but in the middle ages barons did along with his soldiers, servants, jokers, cooks etc.