How do you tell your friend they suck at volleyball me and my best friend want to try out for the volleyball team but she cant even hit the ball and I don't want her to humiliate her self at tryouts?

Let her try out. If she is not very good, she won't make the team, and I'm sure she won't be the only one. It's likely that the ones who'd be watching her at tryouts won't even remember how badly she did, when they have other things to worry about like choosing who to put on the team! If she approaches you afterward about it, you can give her pointers about her game (which will be nicer than telling her she is not good and shouldn't even try). Also, see if your school has a JV team, which tends to accept everyone. Then you both can play and improve without the stress of making varsity.