You like your doctor and he likes you and said he would like to have coffee with you but he doesn't calls he maybe waiting for you to call him first as he is your doctor?

Here is a link to a discussion of the legal and ethical complications for a Doctor Who dates his patient. Your doctor should be aware of them:

http://www.memag.com/memag/article/articleDetail.jsp?id=359038

If you are determined to pursue an outside relationship with your doctor, I'd strongly suggest you transfer to another doctor who is not in the same medical practice or clinic. Due to your own attraction for him, it still would be a good idea to transfer, even if no outside relationship develops.