Which word is correct to use in this phrase the book laid or lain on the table?

Neither. If it's past tense, you would use "lay" (being the past tense of "lie" - intransitive verb).

Present tense: The book lies on the table.

Past tense: The book lay...

Past participle: The book has lain...

Technically, one should not use "laid" (past tense of "lay" - transitive verb) for the said phrase unless someone or something placed the book on the table. "The book which you laid on the table" would be correct.

(This is true unless you subscribe to the folk belief that lie is for people and lay is for things.)