Can you refuse to have the repo man take your car?

You can refuse to release it, but that doesn't mean repossession will stop. Far more detail is necessary to give a more detailed answer.