So imagine you're talking to someone, right? And maybe five minutes after you tell them your name, they forget it. Oh, yeah. Or, you know, you poured your heart out last week about your big life goals, and today, blank stare, like you've never even met. It's kind of like that movie Memento. but for your AI assistant. Exactly. Honestly, that's probably the biggest thing holding AI agents back right now, this goldfish memory problem. Most just have this tiny temporary buffer.
It really stops them from becoming true partners, doesn't it? Or collaborators. Definitely. Welcome to the Deep Dive. Today we're... taking a plunge into fixing that, giving AI real, actual, long -term memory. And it's not just about remembering the last two chat messages, not at all. No, it's much bigger than that. It's about building AIs that genuinely understand you. You know, your preferences, past talks, goals. That's how they give you that really personal evolving help.
So our mission today is to unpack this core problem. We're going to look at how these relational knowledge graphs, something like this open source tool called Zep, can offer a real solution. Zep's basically a database made for AI memory. Exactly. And we're going to walk you through how this memory gets built from scratch. And the big one. How you do it without racking up some crazy huge bill from your AI provider. Yeah, the cost thing is key. We'll also get into some advanced tricks,
the ethics side of things, privacy. Super important. And even give you a kind of four week plan to try this yourself. The goal is you'll understand how to take your AI agents from, well, forgetful assistants to really powerful collaborators. All right, let's get to it. So when you first start with an AI agent, its memory is. Well, pretty basic. Mm -hmm. Very basic. It usually just remembers the last few things said, maybe five to 15 messages back and forth. It's all
in this temporary digital space. If you say, hi, my name is AI Fire, sure, it can say your name back. But, and this is the crucial part, the fatal flaw, really, it's just reading a transcript. The AI isn't learning anything deep. It's just looking back a few lines. So if the chat goes on too long. Or you start a new one later. Yeah. Poof. That old context is just gone. It doesn't understand AI fire as a person. It just knows those words were typed recently. It remembers
the words, not the person. Okay, so contrast that with real long -term memory. What does that look like? Maybe I ask, hey, remind me about that Paris trip plan from last month. Yeah. And the agent, without you feeding in any details again, just comes back with, sure, here's that personalized plan for John Doe's Paris trip next month. Duration. Budget. Around $2 ,000. It pulls John Doe and the budget from its stored knowledge. Not from what I just typed. Exactly, from deep
storage. Learn stuff. That difference, remembering words versus actually understanding a person, that seems huge. What's the fundamental difference there, inside the AI? It's remembering recent chat versus deeply understanding you. Okay, so how does it know all that stuff then? The John Doe details, the budget, the trip. It's this relational knowledge graph you mentioned. Yep, that's the core of it. And it's not just a simple
list of facts. Think of it more like a mind palace, or maybe a really detailed visual map of how knowledge connects. For every single user, it's like their own personal Wikipedia. It stores facts, identifies the key things, the entities, like John Doe or Paris, and crucially, it maps the relationships between them, like John Doe plans trip to Paris. And I guess as you talk
more, that map gets... Dense exactly grows new things get added like maybe the musee door say and new connections like John Doe is interested in art museums Interesting and it's not just the connections each thing in the graph each entity like John Doe or Hotel Paris Central Gets its own little summary generated by the AI something like John Doe user planning Paris trip budget $2 ,000 so this structured brain lets the AI quickly scan for the key ideas and links, makes
its answers way faster and smarter, because it doesn't have to reread tons of old chat logs. So how does this mind map make the AI smarter, essentially? It's a structured brain for quick, precise knowledge retrieval. Let's try to picture this starting from zero. Imagine a totally new user. Let's call him Max. He talks to the agent for the first time. He says, my name is Max. I enjoy hiking, and I currently live in Vancouver, Canada. OK, boom. Instantly, behind the scenes,
Zep kicks in. It creates a main thing, an entity, Max. Right. Then it starts drawing lines, the relationships. Yeah. Max likes hiking. Max lives in Canada. Maybe even AI assistant A's chatting with Max. It's like stacking Lego blocks of data, building up that context piece by piece. And then Max shares more. The graph gets richer. Maybe he adds, I usually hike on weekends. My favorite trail is Grousegrind. I also like taking photos of nature, but I'm not a pro photographer
yet. OK. Graph expands again almost instantly. New lines, Max favorite trail, grouse grind, Max enjoys nature photos, Max skill level amateur photographer, and a new entity pops up, grouse grind, tagged as a hiking trail, Max likes. Okay, now for the payoff, the intelligence test. Max asks, got any ideas for what I could do this Saturday? Right, and the agent doesn't just give some generic stuff, it queries that knowledge graph it just built about Max. It pulls the pieces
together. Just the size, is it? Yeah. The response might be something like, hi Max. Okay, since you like hiking and nature photos, here are some ideas for Saturday. One, hike a new trail, maybe try a different Vancouver trail, mix it up from Grau's Grind, like Lynn Canyon or Cypress Mountain. Ah, see, that's the magic, isn't it? It combined totally separate facts, the hiking, the photos, the location, his usual spot to give a really personalized recommendation. It didn't just recall,
it connected things. Absolutely. What's the magic when the graph grows? Agent combines diverse facts for truly personalized results. So, this all sounds amazing, right? This AI that actually gets you. But, oh, here's the catch. The thing people don't always talk about up front. There's always a catch. As that memory gets smarter and deeper, Every single check can get way way more
expensive. Ah the cost why it comes down to tokens That's how most AI models charge you think of a token as like roughly a word or sometimes part of a word Okay Every time you send a message the agent doesn't just send your message to the big brain the large language model or LLM It bundles up this whole context package. What's in the package usually a summary of you? the user, plus relevant facts pulled from that Knowledge Graph we talked about, and maybe the last few
messages from the chat. Gotcha. So in our simple max example, asking about Saturday, how many tokens was that? Believe it or not, around 2 ,727 tokens, just for that simple question. Now, scale that up. Imagine a loyal customer you've interacted with for months. Their Knowledge Graph might have hundreds of facts. That context package could easily hit 3 ,000, 5 ,000, maybe even 10 ,000 tokens per message. Okay, let's do the math. If it's, say, 0 .002 tons per thousand tokens.
Right. 3 ,000 tokens is 0 .006 cents. Six tenths of a cent. Doesn't sound like much. Quite. But if you have 1 ,000 users chatting daily... That's $6 a day, $180 a month, just for the memory piece. And that can easily balloon into thousands if your graph gets really big or you have lots of users. So why does having a smart AI memory get so expensive? AI models charged by tokens and long context uses many. Right. So the million dollar question, or maybe the thousand dollar
a month question. How do we get this powerful memory without going broke? Method one. Smart context filtering the surgical approach as you called it. Yeah, because a lot of the off -the -shelf memory tools They kind of act like a blunt instrument. They just grab everything all the facts all the history It's frankly often just
lazy engineering. Okay. So what's the fix? You got to take control instead of pulling everything blindly you use direct HTTP requests basically specific web commands to be like a surgeon You precisely select only the relevant info. So you're telling the system just give me the last 10 messages Exactly. Or, show me only the top three facts from the long -term memory that are really relevant to this specific question. Maybe you even add a filter, like ignore anything less than 70 %
relevant. Okay, contrast the flows. Standard flow. User message block. Grab all facts plus entire history directs that stuff it all into the LLM. wasteful, super wasteful, optimized flow, user message block, targeted requests for say last 10 messages, smart search requests for top three relevant long -term facts, merge that small relevant package, then send to LLM. Much leaner, faster, focused. Totally. And you can
get clever with the search query itself. Maybe use the LLM first to refine the user's raw message into a better search term before you even hit the knowledge graph. And I know you mentioned using tools like N8n's code node to handle some of the data formatting. That can be tricky. Oh, yeah. I mean, I still wrestle with prompt drift myself sometimes when trying to get AI to structure complex JSON data perfectly. It's not always
easy. You might ask Claude or another AI to help write the JavaScript snippet to clean it up. That's a good tip. So the results of this surgical approach? Dramatic. Like we saw, you can go from maybe 2 ,700 tokens per interaction down to around 670. Wow, that's a huge drop. Yeah, like a 76 % reduction. Cuts your API costs by more than half. Easy. So how exactly does this surgical approach cut costs so dramatically? By sending only highly relevant filtered data to the AI.
OK, method one sounds great. But sometimes, even being surgical hits a wall, right? You mentioned some APIs make it hard to get history in the right order. Yeah, exactly. Some systems, maybe they give you the whole history, but oldest first. So to get the last 10 messages, you have to pull everything and sort it yourself. Kind of defeats the purpose of being efficient. So that leads to method two, hybrid memory architecture. Right. The best of both worlds approach. The core idea
is super simple, but really powerful. Different kinds of data belong in different kinds of databases. You know, you wouldn't use a hammer for a screw. Makes sense. So what's the two brains set up here? OK, so for your long term memory, that complex web of facts and relationships, you stick with Zep's knowledge graph. That's its superpower. It's built for that. Got it. And for short term. For the recent conversation history, the last 10, 20 messages, use a standard Simple database
like Postgresql. It's super fast and really efficient for just storing ordered lists with timestamps. Perfect for recent chat history. Ah, okay. So you get the deep understanding from Zepp, but the lightning fast recall of recent stuff from Postgresql. Exactly. The best of both worlds. How does the flow work then? Message comes in. Message arrives. Then at the same time, you fire off two requests. One API call to Zepp for the
top. say three relevant long -term facts, and D, a super quick query to PostgreSQL for the last 10 messages. Okay, parallel requests. Yep. Then you quickly merge those two small relevant context packages together, send that combined package to the LLM. And after the LLM replies? You update both memories. Add the new exchange to PostgreSQL and let Zep process it to update the knowledge graph if needed. Let's picture it. User asks something complex like, where should
I move? I need a place that fits my interest. Right. An agent with this hybrid setup can give a really nuanced answer. It uses ZEP, the long -term brain, to pull facts like enjoys hiking, knows about Lynn Canyon Park, interested in photography, and it uses Postgresco, the short -term brain, for the immediate context, like the user just mentioned, my future, or thinking about change. It combines both for a personalized insight without wasting tokens on stuff that's not relevant right
now. What's the main advantage of this hybrid two -brain approach, then? It combines deep relational understanding with lightning -fast recent recall. This is where it gets really exciting. Moving from a single demo to something that works for tons of users. The key is session IDs, right? Absolutely. It's fundamental. The whole system hinges on this. Every unique user gets their own unique identifier, the session ID. Think of it as the key to their own private knowledge
graph. So this could be their Telegram chat ID. Or their email address if it's an email bot or user account ID from your website. Anything unique to them. And the benefit. Massive scalability. Thousands, even millions of users can be talking to the agent at the same time. But each conversation is totally separate. Max's knowledge graph doesn't leak into mics. It's completely isolated. That's how you go from a chat bot to a real AI workforce. Think about the applications. Customer support.
Oh, yeah. An agent that remembers your entire purchase history, pass support tickets, even your preferred way of communicating. You never have to repeat yourself. It feels like talking to a dedicated account manager who actually knows you. Or an educational tutor. Building a unique learning profile for every single student. Tracking their progress, spotting weaknesses, adapting teaching styles automatically. It's incredibly
powerful. The sales assistant. Imagine. Detailed history for every prospect needs, objections raised before, personal interests they mentioned offhand. It's the perfect briefing for the human salesperson stepping in to close the deal. And even just onboarding new users. Remembering exactly where they left off in a complex setup process that could massively reduce churn. People hate starting over. Whoa. Just imagine the impact. A sales agent that knows hundreds of prospects
intimately, instantly. or a support agent recalling every single detail. That really scales human -like intelligence. It's a total game changer for the user experience. So how do these AI agents manage to remember so many different users distinctly? Each user has a unique session ID and a private knowledge graph. Okay, so we've got the core methods down. What about taking it to the next level? Pro strategies. Definitely things you can do. For cost optimization, you can play with
dynamic relevance scoring. Meaning? Adjusting that relevance threshold. Maybe lower it for creative brainstorming tasks where you want more tangential ideas, but crank it higher for technical support where accuracy is paramount. OK, that makes sense. You can also set up entity prioritization. Tell ZEP, hey, things like pass support ticket ID are way more important than favorite color. So it prioritizes retrieving the critical stuff. Nice. What about old information? Intelligent
memory decay. Basically, if a fact hasn't been relevant or accessed in a long time, its important score gradually fades. It stops outdated info from cluttering things up. Smart. And for really big scale. You might look beyond just Postgresql for the short -term memory. Maybe Redis for absolutely blazing fast caching if you have users hitting the same topics repeatedly. Or Elasticsearch if your knowledge graphs become truly enormous. And managing the memory itself. Critical. You
need limits. Set a max number of facts per user graph. Have a strategy to archive graphs for inactive users on a cheaper storage. Otherwise, costs and slowdowns are inevitable. It sounds like there are pitfalls, too. What are the common mistakes people make building these? The landmines to avoid. Oh yeah, plenty. Number one is probably over -storing. Saving every tiny detail makes the graph noisy and less useful. Use those relevance thresholds. Prioritize entities. Session ID collisions.
Using ideas that aren't truly unique. Disaster. You mix up user data. Always use secure, unique IDs like UIDs or properly hashed identifiers. Unbounded memory growth. Just letting graphs grow forever without limits or archiving leads to slowdowns, ballooning costs, implement those limits. Poor relationship quality. Sometimes the AI extracting facts, it gets it wrong, creates weird or incorrect links like Maxi located in
Panama when he's in Vancouver. You need to validate, maybe fine tune the prompts used for extraction. Good point. And the last one. Ignoring token optimization. Just assuming memory is worth any cost. You have to monitor usage and implement filtering like we discussed. Costs can sneak up on you fast. So what are the biggest mistakes people make when building AI memory systems? Over storing data, wrong IDs, unchecked growth,
poor data quality, ignoring costs. Building these powerful memory systems isn't just tech though. There's a huge ethical dimension. Absolutely. With great memory comes great responsibility. A system that remembers so much about someone requires you to be a really careful guardian of their privacy. What are the key principles there? Transparency, number one. Tell users the agent remembers things to help them. Something simple like, to improve our chats, I'll remember
key details. Goes a long way. And letting users control their data. Crucial. The right to be forgotten, like under GDPR. Users must be able to easily see their data. export it, and most importantly, delete it if they want to. And security, obviously. Non -negotiable. Especially if you're storing anything remotely sensitive, robust security is a must. And you mentioned performance benchmarks earlier. These optimizations aren't just theory, they have real impact. Huge impact. We looked
at cost per 1 ,000 interactions. Basics app might be, say, $150 to $240. Okay. Our optimized HTTP filtering method. cuts that way down maybe $60 to $90. Big improvement. But the hybrid architecture, that's the winner. We saw a cost between $48 and $72. That's a massive saving compared to the basic setup. And user experience improves too. Dramatically. Response times drop from like three eight seconds down to one three seconds. Accuracy jumps from maybe six out of ten to eight
point five out of ten. User feedback goes from yeah it's okay to wow this thing actually gets me. And it's fast. So what's the biggest tightrope walk for developers building these powerful AI memories? Balancing advanced memory capabilities with user privacy and data security. OK. So for listeners who are thinking, all right, I want to build this, you put together a kind of four -week plan. Yeah, a practical roadmap to get started. Week one, foundation setup. Meaning?
Get Zepp installed. Get PostgreSQL running. Maybe grab a pre -built workflow template. Start having simple chats just to see the graph begin to form. Get the basics working. Week 2. Customization. Start tailoring it to your specific need. Adjust those relevant scores. Maybe define some custom types of entities you care about. And critically, set up proper session ID handling for how your users will connect. Week 3. Optimization. Now, implement those cost -saving tricks. Fine -tune
the limits, the relevance filters. Double -check those PostgreSQL queries are running fast. Set up monitoring so you can actually see your token usage and costs. And weak for. Production deployment. Test it hard with simulated users to make sure everyone's data stays separate. Set up backups and a recovery plan. Then you're ready to go live. So the bottom line here, this isn't just
about fancy or chatbots. No. Not at all. This is really the foundation for a whole new class of AI agents, agents that can actually form genuine, useful long -term relationships with people. They learn from the past, they understand what makes each user unique, and they provide value that actually gets better over time. Right. The core message is simple. The smartest AI model in the world is basically useless if it can't remember what actually matters to the person
it's talking to. And building this kind of memory, this understanding, that's a real edge. Huge competitive advantage. While everyone else is building agents that forget everything tomorrow, you can build agents that learn and grow with your users. It's about remembering what matters, doing it securely and doing it respectfully. So here's a final thought to chew on. As AI gets more and more woven into our lives, how do you
think our own human memories might adapt? when we can lean on these agents that, well, never forget. That's a deep question. And what does understanding even mean when an AI can perfectly recall every single thing you've ever said to it? Lots to think about there. We definitely encourage you to explore these possibilities, giving your AI agents memory that's intelligent, affordable, and ethical. That's our deep dive for today. Out, T -Pro Music.