Is dating a married man wrong even if he is not with his wife?

That depends on what your standard of right and wrong is. If your standard is what The Bible says, then yes, it is wrong to date a married man. The Bible says that a man is to "cleave unto his wife" only, meaning he is not to date or marry or have sex with any other woman.

Genesis 2:24 Therefore shall a man leave his father and his mother, and shall cleave unto his wife: and they shall be one flesh.