Better Text Generation With Science And Engineering

Current text generators, such as ChatGPT, are highly unreliable, difficult to use effectively, unable to do many things we might want them to, and extremely expensive to develop and run. These defects are inherent in their underlying technology. Quite different methods could plausibly remedy all these defects. Would that be good, or bad?

https://betterwithout.ai/better-text-generators

John McCarthy’s paper “Programs with common sense”: http://www-formal.stanford.edu/jmc/mcc59/mcc59.html

Harry Frankfurt, "On Bullshit": https://www.amazon.com/dp/B001EQ4OJW/?tag=meaningness-20

Petroni et al., “Language Models as Knowledge Bases?": https://aclanthology.org/D19-1250/

Gwern Branwen, “The Scaling Hypothesis”: gwern.net/scaling-hypothesis

Rich Sutton’s “Bitter Lesson”: www.incompleteideas.net/IncIdeas/BitterLesson.html

Guu et al.’s “Retrieval augmented language model pre-training” (REALM): http://proceedings.mlr.press/v119/guu20a/guu20a.pdf

Borgeaud et al.’s “Improving language models by retrieving from trillions of tokens” (RETRO): https://arxiv.org/pdf/2112.04426.pdf

Izacard et al., “Few-shot Learning with Retrieval Augmented Language Models”: https://arxiv.org/pdf/2208.03299.pdf

Chirag Shah and Emily M. Bender, “Situating Search”: https://dl.acm.org/doi/10.1145/3498366.3505816

David Chapman's original version of the proposal he puts forth in this episode: twitter.com/Meaningness/status/1576195630891819008

Lan et al. “Copy Is All You Need”: https://arxiv.org/abs/2307.06962

Mitchell A. Gordon’s “RETRO Is Blazingly Fast”: https://mitchgordon.me/ml/2022/07/01/retro-is-blazing.html

Min et al.’s “Silo Language Models”: https://arxiv.org/pdf/2308.04430.pdf

W. Daniel Hillis, The Connection Machine, 1986: https://www.amazon.com/dp/0262081571/?tag=meaningness-20

Ouyang et al., “Training language models to follow instructions with human feedback”: https://arxiv.org/abs/2203.02155

Ronen Eldan and Yuanzhi Li, “TinyStories: How Small Can Language Models Be and Still Speak Coherent English?”: https://arxiv.org/pdf/2305.07759.pdf

Li et al., “Textbooks Are All You Need II: phi-1.5 technical report”: https://arxiv.org/abs/2309.05463

Henderson et al., “Foundation Models and Fair Use”: https://arxiv.org/abs/2303.15715

Authors Guild v. Google: https://en.wikipedia.org/wiki/Authors_Guild%2C_Inc._v._Google%2C_Inc.

Abhishek Nagaraj and Imke Reimers, “Digitization and the Market for Physical Works: Evidence from the Google Books Project”: https://www.aeaweb.org/articles?id=10.1257/pol.20210702

You can support the podcast and get episodes a week early, by supporting the Patreon: https://www.patreon.com/m/fluidityaudiobooks
If you like the show, consider buying me a coffee: https://www.buymeacoffee.com/mattarnold
Original music by Kevin MacLeod.
This podcast is under a Creative Commons Attribution Non-Commercial International 4.0 License.