AI Agents Have a Memory Problem (And You're Probably Making It Worse)

Coding Chats Episode 79 - Richmond Alake, Director of AI Developer Experience at Oracle, joins John to discuss agent memory — how AI agents store, retrieve, and adapt to information. He argues that developers building memory on flat files are naively reinventing the database, and that once you factor in concurrency, security, and scalability, a proper database is inevitable. The conversation covers the full memory stack and how Oracle's AI database keeps embeddings and data together without shipping sensitive information to external providers.

The pair also explore why memory is the most universally relatable concept in AI, the history of how neuroscience shaped LLMs, and the problem of Catastrophic Forgetting that still haunts models today. A sharp AGI debate lands on a sobering point: an LLM is just a function — tokens in, tokens out — and most AI engineers are unknowingly rediscovering solutions that database engineers spent decades building.

Chapters

00:00 — What Is Agent Memory and How Does It Work?

05:00 — File System vs Database: Which Should You Use for Agent Memory?

09:00 — Why Building on Files Means You'll Reinvent the Database

13:00 — How Oracle Is Meeting AI Developers Where They Are

15:00 — Why Memory Is the Most Universal Concept in AI

21:00 — From Computer Vision to LLMs: How Richmond Found His Path

24:00 — Catastrophic Forgetting: The Problem That Hasn't Gone Away

26:00 — Is AGI Real? Why the Goalposts Keep Moving

33:00 — Handling PII, Data Sovereignty, and Access Control in AI Apps

42:00 — The Rise of Memory Engineering: AI's Most Underrated Discipline

Richmond's Links:

LinkedIn: https://www.linkedin.com/in/richmondalake/

X: https://x.com/richmondalake

John's Links:

John's LinkedIn: https://www.linkedin.com/in/johncrickett/

John’s YouTube: https://www.youtube.com/@johncrickett

John's Twitter: https://x.com/johncrickett

John's Bluesky: https://bsky.app/profile/johncrickett.bsky.social

Check out John's software engineering related newsletters: Coding Challenges: https://codingchallenges.substack.com/ which shares real-world project ideas that you can use to level up your coding skills.

Developing Skills: https://read.developingskills.fyi/ covering everything from system design to soft skills, helping them progress their career from junior to staff+ or for those that want onto a management track.

Takeaways:

File systems are fine for prototyping, but the moment you hit production scale you're just slowly reinventing the database.

File systems are fine for prototyping, but the moment you hit production scale you're just slowly reinventing the database.

Agent memory isn't a new concept — it's data management, and database engineers have been solving it for decades.

Memory is the single most relatable entry point for explaining AI to anyone, technical or not.

Catastrophic Forgetting isn't a solved problem — it plagued RNNs and still quietly haunts LLMs today.

An LLM is ultimately just a function: tokens in, tokens out — which should temper any claims about sentience or AGI.

The definition of AGI keeps shifting to match whatever AI can't do yet, making the whole debate almost meaningless.

Most AI engineers have less than ten years of experience and are unknowingly rediscovering solutions that search and database engineers spent decades building.

"Vector search is all you need" is one of the most dangerous oversimplifications in AI engineering right now.

Memory engineering — the crossover between data engineering, search optimisation, and agent design — is an emerging discipline that doesn't have a name yet but absolutely should.

The real moat in AI products isn't the LLM itself, it's everything built around it — the harness, the memory, the retrieval pipeline.