Have you ever spent weeks engineering an AI agent only to have it confidently deliver an answer that was just demonstrably catastrophically wrong? It's the paradox of this current AI generation, isn't it? The delivery is perfect, but the information is just terrible. You ask for Q4 revenue, you get a number off by a factor of 10. Or you ask for a contract summary, and it misses the one single liability clause that matters. Exactly. It's a hallucination, but it's delivered with
this conviction of a seasoned expert. Welcome back to the Deep Dive. Today, we are tackling that exact engineering frustration. For anyone out there building agents, the secret we found is that the flaw, well, it's rarely the language model itself. No. The problem is almost always rooted in how we feed the data to our systems. Which is what we're calling context engineering.
Precisely. It's the whole process of ensuring your retrieval augmented generation, your RAG system, has the exact information in the right format at the precise time it needs it. So RAG is basically giving your LLM some external knowledge to keep it grounded. Yes, and our mission today is to help you transform your agents from these confident guessers into truly verifiable analysts. OK, let's unpack this. We need to move beyond those intro tutorials that always seem to lead
to these frustrating failures. For sure. We're going to cover four powerful, specific solutions. We'll start with a deep dive into why the basic default methods fail so badly. Then we'll get into the reliable methods, starting with structured data simple filters and SQL queries. And finally, we'll define when you should actually use full context retrieval or when that traditional vector search is still the right tool for the job. Let's
do it. All right. So when you first start building these RIG systems, the default method everyone learns is chunk -based retrieval. It's usually powered by vector search. Great. You take a massive document, you cut it up into these little pieces, these chunks. And then the AI only gets to see a handful of them. maybe the three or four things are most relevant. It's like stacking scattered Lego blocks of data and just hoping they form
a perfect blueprint. That standard, that fragmented approach, it basically guarantees three critical issues. The first one is the most common, missing context. The little fragments, they might contain keywords, sure, but they lack all the surrounding narrative. The setup, it's like, if you were summarizing a 100 -page book and I only gave you pages 12, 45, and 88. You'd have no idea what's going on. You couldn't possibly grasp the main character's motivation. You'd miss the
crucial setup from chapter one. The LLM, when it's faced with these disconnected pieces, it's just forced to guess. Yeah, and I can offer a vulnerable admission here just based on my own early failures. The second issue is what I call the math wall. Oh, yeah. Language models are, I mean, they're incredible text processors, but they are fundamentally terrible at complex, verifiable math. They're calculators that are bad at math.
Exactly. I once asked my internal agent for Q3 sales figures from this sprawling 2000 line report. It pulled four records that happened to contain the word total. And it just added those four up. It summed those four lines and presented that tiny result. as our company's entire quarterly revenue. And that is the core absurdity, right? The agent calculated just a tiny fraction of the necessary data, presented the result with
99 % confidence. And you have to go use external tools just to realize it's completely broken. It happens because that chunking process, it prioritizes what sounds similar over actual data integrity. Right. And that leads to the third flaw, which is the problem of bad summaries. Yeah. So you need a comprehensive chronological summary of a 50 page technical manual. If your system just randomly hands the AI four paragraphs from all over the document. The summary is going
to be useless. It's like trying to write a detailed review of a three hour documentary after only watching like 30 seconds of the trailer. You lose the plot. the sequence, everything. So if we accept that these math errors and logic gaps are so easily created by fragmentation, how should a developer even approach verifiable accuracy at this point? Well, math errors are just inherently masked by the LLM's confidence. You have to force the calculations onto external specialized computation
engines. OK. So let's pivot. Let's talk about the first real solution, moving away from all this guesswork. Right. And this one is specifically for structured data. So data that lives in neat rows and columns, think an internal dashboard or even just a Google sheet. Instead of asking the AI to search and guess, we teach it to operate more like a database admin. We teach it to filter. And this approach relies on what's called function calling, or defining these very specific tools
the AI can use. We don't ask it to read the data. We ask it to construct a precise query. So for a sales database, for example, we define functions like product name query or date query. And the technical depth here, the really important part, is defining that tool's specification. You have to dictate the exact inputs. So for instance, requiring the date to be strictly in that yyymmdd format. That input validation is so crucial.
It is. And we had to give the agent a core system prompt, an instruction that says, always use the most specific tool available for the user's intent. And that instruction, it turns the agent into this highly disciplined engineer, not some creative writer trying to guess what you mean. Exactly. So consider a real -world query. How many wireless headphones were sold in the Northeast region on 2025 -0915? The agent doesn't search a vector database. It sees the query. It identifies
the need for filters. And it just chains the tools together. So it uses product name query, then region query, then the date query to narrow everything down. And then finally, it hands that small, perfect data set to a dedicated calculator or computation engine to just sum the results. The outcome is 100 % accurate. Because the heavy lifting was done by precise data filtering, not by linguistic guesswork. Right. But I should say, for listeners who might not have a deep
programming background. Yeah. The simplicity of that outcome, it really masks the complexity of the setup. I mean, defining those tools and making sure they don't suffer from prompt drift, where the agent just starts ignoring your instructions over time. I still wrestle with that myself. It's a constant battle. It absolutely is. So what makes implementing these specific filters superior to just using generalized chunking,
even for really large structured data sets? Filters provide targeted data access, which eliminates the risk of data fragmentation and guarantees 100 % verifiable calculation inputs. OK, so filters are great for selection. But what if you need more complex calculations, things like generating averages or standard deviations or sorting complex reports. Yeah, the AI is going to fail those manual calculations every single time. And this is where we bring in the language of databases.
SQL. So think of this as giving your ARAGAY LLM a superpowered calculator that's optimized for massive data operations. Exactly. The process is pretty elegant. The LLM writes the SQL code, maybe a complex group PO by Y function for a Postgres database, sends that query to the actual database engine. And the engine just returns the single precise result. But the critical engineering detail. The part you cannot get wrong is in the system prompt. You have to include the exact
table schema. So you have to explicitly tell the AI, hey, there's a table named sales data, and then list every single column. Customered, INT, total price, float, order date, date. Because if you omit the exact column names or the data types, the AI just guesses. And that leads to immediate execution failure. Right. And that's why this is a step up in complexity from simple filtering. You're actually relying on the LLM to write code. And you have to instruct it with
absolute clarity. Never attempt math manually. You force it to delegate the calculation using functions like sum, abg, and group pby. You're pushing all that mathematical load onto the database. That raises a critical point, though. If we're having an LLM write code, doesn't that open us up to huge security concerns like SQL injection vulnerabilities? Isn't it much slower than just fetching the data? That is a crucial technical challenge. Yes, the developer has to implement
really strict input sanitization. And you typically have to limit the types of queries the LLM can even construct to prevent malicious injection. But you use SQL for speed on complex aggregation, not simple filtering. When a user asks, who are our top three customers by total spending this year? The SQL script processes millions of rows instantly. Ah, OK. So you need that power when a simple filter just won't cut it for the complex
sorting and counting. Exactly. We let the database do the heavy lifting for absolute reliability. So why does providing the specific column names and data types prevent the most common SQL failures in these agents? Without the exact schema, the AI guarantees query failure by producing type mismatch errors or ambiguous column references. Makes sense. Now let's move back to unstructured data. We're talking about those long legal contracts, technical manuals, or sprawling video transcripts.
Sometimes you really do need the AI to understand the entirety of a source, not just little fragmented parts. modern large context windows, like 128k tokens, this idea of full context retrieval is not only viable, but it's often the most accurate option. Right, you're giving the model a huge memory, the full text. But there are two different ways to do this and understanding the cost differences, well, it's key. Method A is the simple prompt method. You just copy and paste the entire document
into the AI agent's static system prompt. But the cost drawback of that simple method is immediate. And it's heavy. Yeah. Since that long text is just hard -coded into the prompt, you pay for every single one of those tokens, the entire document, every single time the agent runs, even if the user just asks, hello, how are you? You're just burning tokens for no reason. Exactly. And that's why we really recommend method B, the tool -based method. We wrap the entire document
inside a tool definition. It's the difference between forcing an employee to carry every file in their arms all day. versus just giving them a key to the filing cabinet. Maximum savings, on -demand access. The agent only fetches and reads that full, expensive document if the user's query specifically triggers that tool. So if the user asks about the document, the agent gets the full content instantly. But if they ask about the weather, it saves the cost. And the performance
increase is just stunning. We saw a test where standard vector search created this messy chronological nightmare from a video transcript. But full context. Because it had the entire script, it produced a perfect sequential and comprehensive summary. So if accuracy on a single document is paramount and it fits within that context window, this is the choice. Whoa. Imagine scaling this precise chronological retrieval to a billion queries across complex legal documents without losing
a single key detail. The analysis potential is just immense. It guarantees context continuity. So given the cost and the speed trade -offs, why should we always avoid that simpler prompt method for long documents? Avoid the prompt method because you will incur unnecessary token cost when the user's query does not require accessing the document. Okay, so... We've been a little tough on chunk -based retrieval, but let's be absolutely clear. Vector search is not a bad
technology. Not at all. It's just deployed inappropriately, like 90 % of the time. Vector search or chunk -based retrieval, it remains the undisputed king for one specific thing. Massive scale. It's the library card catalog analogy, right? You use vector search when you have 10 ,000 documents, a massive library of employee handbooks, old support tickets, internal memos. You simply cannot afford to feed the AI all of them using full context, and they're way too unstructured for
filters or SQL. And the beauty of it is the indexing. It embeds all that text into these numerical representations, stores them in a vector store. And can rapidly retrieve the pages that are numerically closest to the user's question. It helps you find the right book or the right page within the right book really quickly. So the distinction is crucial. You use vector search when the answer could be anywhere in this sprawling library. It's amazing for the find the needle in the haystack
kind of query. But it's fundamentally terrible for analytical tasks. Things like summarize everything or count the total revenue. So of all these powerful context methods, what unique capability does vector search offer that the others just can't replicate? VectorSearch excels at rapidly indexing and retrieving relevant information from massive sprawling libraries of unstructured documents. Okay, so that brings it all together. How do we decide which of these four methods to use
in a real -world scenario? Well, the decision framework is actually incredibly simple. We should always start with this question. How would a human knowledge worker answer this exact question? OK. So if a human would naturally use an Excel filter, a query like, show me all orders from yesterday where the product was blue, then you
have to use filters. Right. And if a human would need a pivot table or a complex calculator, a question like, what's the average lifetime spending of our top 10 % of customers, then you must use SQL queries. And if a human would need to read the whole file sequentially, like summarize this 40 page contract, noting all termination clauses,
they use full context retrieval. And finally, if a human would need to search through a massive filing cabinet, if the answer to, how do I reset my password, is buried among thousands of old user guides, then you use vector search. Okay, but what about the layered complexity? What if you get a query like, summarize the Q1 marketing strategy document for new customers in Texas who spent over $500? That is where context engineering really earns its name. the agent has to chain
these things together. First, it uses the filter tool to identify the relevant customers. Texas greater than $500. Then it uses the SQL tool to aggregate and sort their spending trends. And then finally. Finally, it uses the full context retrieval tool to fetch and summarize that single marketing strategy document that was mentioned in the query. It's like a chain of custody for information. That layered approach ensures you
get maximum accuracy at every single step. So finally, maybe a few essential technical warnings. Yes. Don't be lazy with your system prompts. Give your tools descriptive, specific names that tell the agent exactly what they do and what data format they require. And never ever combine structured data like numbers, dates, or precise inventory counts with unstructured text. blindly in a vector database. Keep that structured data
separate. If you mix your math with your text, you are guaranteeing the mathematical analysis will fail. That feels like the core lesson here. I think so. The main takeaway we've distilled today is that building reliable RRAG LLMs, it isn't about finding the perfect model. It's all about context engineering. It's about ensuring the AI has the exact, verifiable information in the right format at the precise time. We move past all the guesswork and into reliable, accountable
analysis. And remember, the best agents are polyglots. They use a mix of these methods. You might filter to narrow down a custom race, then use full context to read their contract, then use SQL to report on their spending. That layered accuracy really is the new standard. So here's your final thought for reflection. Look at your current AI workflows. Ask yourself if you're using that generalized vector search where a simple precise filter or a structured SQL query would give you 100 % more
accuracy and auditability. Try changing just one tool in your pipeline this week. Go build something amazing with this knowledge and we look forward to the next deep dive with you.