Why won't he talk about the affair and refuse to even admit it?

If he is not willing to admit that is an obvious answer that he is guilty. Unless, you have continuously falsely accused. Of course if he is guilty he does not want to talk about it! Men will deny it until they die because they dont want to lose their options. You are an option for them, incase it does not work out with the other person they still can be with you. If they admit it they know that they have lost you permanently.