Is ball bag a swear word?

It is what you would call a "vulgarism". Not really obscene (swearing), but not polite language, either.