What stops one branch of the government from being too powerful?

Checks and balance, separation of power.

See: Executive branch, Legislative branch, Judicial branch.

____________________________________

However, this system only works when the other two branches of government use their OWN powers to prevent the out-of-control branch from amassing power.

For example, if the Executive branch tries to claim too much power, the Legislature has the power to impeach and remove the executive branch member who is out of control. Impeachment was never intended to be reserved for obviously criminal activities, but also for unconstitutional abuses of power. Impeachment is an inherently POLITICAL act. The power of impeachment has been re-interpreted so narrowly that it is essentially nullified.