What does 5o in the house mean?

The 5-O are the police, so the phrase is a warning that there are cops around or there is a police raid in progress. People shout this in rough areas to give folks time to run, flush/hide any contraband, or whatever. "In the house" usually refers to something being in the current location.

This slang term for police comes from the title of a television show, Hawaii Five-O. In the original name, the O was a zero instead, in honor of the fact that Hawaii was the 50'th US state. Since Hawaii Five-O was a police show, the term "five-O" became associated with the police.