What does he mean when he says I dont want to be with you you need to be with you?

Probably means that you need to work out things by yourself, like 'alone time'. Or it could mean he/she thinks you're too clingy. Hope this helps.