Can you be allergic to roughage?

Roughage (more or less synonymous with dietary fiber) is generally plant based, but it's specifically non-digestible. Since it doesn't break down and get absorbed, it's not going to a good candidate to cause allergic reactions. That said, a food (bean, grain, vegetable) that provides roughage will generally have enough other digestible elements that they can trigger a reaction. For instance, the skin of a tomato is high in fiber/roughage, and that fact won't be what causes an allergic reaction; you might simply be allergic to the other compounds in a tomato. As an analogy, it's kind of like asking if bottles are dangerous to drink from; if they have 100% alcohol in them, then yes. If they contain juice, then no. It's not really about the bottle.

Last point: you could definitely have a bad physical reaction to fiber without it being an allergy per se. For instance, if you're sensitive to it, you could easily get diarrhea.