Is the driver or the owner legally responsibile in an accident?

Insurance stays with the car. So if the vehicle is insured (subject to any policy exclusions) and your 'permissive driver' were At Fault then your policy will be primary on all damages. You both however are 'legally' responsible. If you are saying that neither the vehicle nor the driver had insurance. Then you both have a problem, and more than likely the injured party (or their insurance) will come to both of you for reimbursement.