Went is simple past, whereas gone is an ed-participle (traditionally called past participle).
'I haven't gone' is in the perfect aspect, which takes ed-participles.
(Note that certain dialects take 'I haven't went', but it's considered non standard.)
Here is a simple way to think of it as well.
"Went" is past tense, something that has already happened--you can't really say you haven't done something that has already happened. It doesn't make sense.
Other examples include Ate & Eaten.
Ate is something concrete that you have already done, or already has happened.
"I haven't ate yet," is incorrect, unless you have a time machine.
"I haven't eaten yet," is correct.
I haven't spoke <--incorrect.
I haven't spoken
