What does it mean to write a penultimate novel?

Penultimate means next to last.

So a penultimate novel is the next to last in a series of novels eg the second novel in a set of three (a trilogy).

Contrary to popular belief, the term "penultimate" (despite that it contains the Word "ultimate") has nothing to do with the "best", "highest" or "last". It means the 2nd to last; as such, a "penultimate novel" would be a novel that is written just before the last novel, i.e. in the collection of 7 children's novels by C.S. Lewis called The Chronicles of Narnia, the penultimate novel would be The Magician's Nephew (#6), which comes before the last novel, The Last Battle (#7) in the popular series.