How do you say 'It is fun to make gingerbread houses with my family' in French?

Well, gingerbread is pain d'épices, and make houses is bâtir des maisons, and so you might say C'est amusant de bâtir des maisons en pain d'épices avec ma famille, but no one is likely to know what you mean without a lengthy explanation.