Can you take Ativan while on clomid?

I was talking Clomid and Zoloft the first cycle and did not get pregnant. Talking more to my doctor she said to stop taking the Zoloft. The pharmacy says that Zoloft lowers your ability to get pregnant.