Is it legal for my next door neighbors to secretly record me in any part of my backyard with a surveillance camera in California?

i wouldn't say so.

ANSWERI would say it's an infringement on your right to privacy, ESPECIALLY if it's pointing in one (or more) of your Windows. Talk to a lawyer, or possibly even call the police department/sheriff's department in your area. (In other Words, I believe it to be ILLEGAL for them to be doing this)