I've had several thoughts on the Carpathian Dwarkesh conversation that took place in late October of 25, but the one that keeps haunting me is something Karpathy just kind of casually mentioned before moving on to another topic. I think it might be the biggest idea in the whole conversation. He was talking about human model similarities and he says humans collapse during the course of their lives. Children have an overfit, yet they will say stuff that will shock
you because they're not yet collapsed. But we adults, we end up revisiting the same thoughts. We end up saying more and more the same stuff. The learning rate goes down, the collapse continues to get worse, and then everything deteriorates. End quote. Since my 20s, I've been terrified of this happening to me. It pierces my soul whenever my partner says things like, I knew you were going to say that. Ouch.
Predictable humor or wit isn't another example. How many older people do you know who tell the same stories and jokes over and over? They watch the same shows. They listen to the same five bands, and then eventually, like 2 or 1, their aperture slowly shrinks until they die. Luckily, Karpathy gives a solution right after we have to find sources of entropy. When we were kids, everything was entropy because everything was new. So we were constantly changing our preferences,
our behaviors, our language and everything. It made us fresh, unpredictable, which is highly related to the concept I'm obsessed with from Shannon's theory of information, which in his model defines information as the part of transmission that isn't repeated or noise. I think about this constantly when I'm giving talks or participating in panels or whatever, or when I'm watching someone else do so. The main thing I'm asking myself, especially for my own content, is how much of this is new?
How often will I'm presenting this? Will the viewer be pleasantly surprised? If the answer is not very often I redo it or I start over. I'm actively doing a bunch of stuff in addition to pathological reading to maximize entropy in my life. I'm reading a lot of old books on writing, like rhetorical figures and stuff like that, to try to get fresh phrases into my mind. I regularly reread and listen to Christopher Hitchens books and debates.
Just having exposure to that level of non cliché language. And I'm currently building in cloud code a skill called Increase Entropy that incorporates all of this old and fresh language like a particle accelerator. So I can point it at a thought or a piece of content and basically come up with novel ways of saying the same thing. So I give it the way that I would say it in a kind of like just breaks me out
of my mold. I even went so far in 2024 to create an AI prompt in fabric that would rate talks, blogs, panels, or whatever for wows per minute, meaning how often a given piece of content surprised the audience. I mean, this was a problem before AI, and now many are delegating even more and more of their thinking to a system that learns by crunching mediocrity from the internet. I can see things getting way worse in this respect. I guess it's somewhat comforting that this happens to both AI models
and to people. It makes the whole thing more human somehow. And hearing Karpathy say it so plainly was jarring to me in a pleasant way. At least for us humans. The solution seems something like recognize that this is a problem that starts for everyone in there, probably like mid to late 20s, and constantly seek and consume sources of novelty and freshness to maintain young mind.