Where does the saying that's the pits come from?

I would think it meant the "pit" of a fruit, the part you throw out (unless you plan on planting it). The fruit is the eatable yummy part, the pit the uneatable yucky part.