Can semen do damage to you if swallow frequently?

No, it is not harmful, in fact there is research that indicates that semen can help prevent Breast cancer.

Scientist are shocked that breast cancer could be reduced 50% if all women had regular, three times a week, semen exposure; like Semen therapy

Characteristics of reproductive life and risk of breast cancer in a case-control study of young nulliparous women

http://www.sciencedirect.com/science/article/pii/0895435689901212

Barrier contraceptive practice and male infertility as related factors to breast cancer in married women

http://www.sciencedirect.com/science/article/pii/0306987778900518

Barrier contraceptive practice and male infertility as related factors to breast cancer in married women. Preliminary results.

http://www.ncbi.nlm.nih.gov/pubmed/566883

And swallowing healthy semen three times a week has other great benefits too; longer life span, slower ageing, clear skin, burns excess fat, prevents Diabetes, elevates mood, natural high, prevents depression and prevents the donor getting prostate cancer.

- SemenTherapy.com