The Pre-Training Wall and the Treadmill After It
May 09, 2026•56 min•Ep. 121
Listen to this episode in Metacast mobile app
Don't just listen to podcasts. Learn from them with transcripts, summaries, and chapters for every episode. Skim, search, and bookmark insights. Learn more
Episode description
I've been confusing Don with frontier-lab links late at night for a bit.
Ilya Sutskever told a NeurIPS audience that pre-training as we know it would unquestionably end. There's only one internet, and the data isn't growing. The frontier labs call this the pre-training wall.
A leaked Google memo from 2023 argued they had no moat. R1 is on GitHub. Llama is on Hugging Face. OpenAI's secondary-market valuation has climbed past $850 billion.
Don was confused. So he came over and we made an episode about it.
For the best experience, listen in Metacast app for iOS or Android