Is there a such thing as being allergic to white people?

No, this is a figure of speech to say you don't like white people (racism)

There is a medical condition called albophobia which is phobia of white people.