Is there a limit to the number of troops in a space marine army?

The only imposed limit is that of the maximum points value of the game you're playing. So long as the combined points value doesn't exceed the stated max, you can have as many actual troops as you want.

If you want to be ultra strict though, you could never have more than 1000 troops, for codex marine legions.