Can a teacher search a Google hangout on a students personal Google account and use messages found as evidence against another student?

It depends on the school's policies. At my daughter's school, they sign an anti-bully agreement at the beginning of the school year. It advises that the school can and will use online posts when considering bullying or criminal accusations. It is best to check with your individual school district to see what their policies are.