Who are congress members primarily responsible to?

Their constituents* and their political party (mostly democrats or republicans)

*Constituents are the people that members of Congress Represent.