What is the limit for the deck joist that cantilever to overhang?

Good rule of thumb (and most code books) state that the maximum overhang should be no more than one-third the total length of the joist. This provides two-thirds of the joist as support.