Who said Never have children only grandchildren?

This quote is often attributed to Eugene Luther Gore Vidal (October 3, 1925 - July 31, 2012). However, it was actually his maternal grandfather, Thomas Pryor Gore, that said it. Thomas was a US Senator. He is known for other similar quotes such as, "Be not fruitful, do not multiply."