Imagine this. You tell a really powerful AI that a new, maybe fictional, alloy, let's call it adamantium -7b, melts at 3200 degrees Celsius. You've given it this fact right there in a document. Then you ask it, okay, according to this document, what's its melting point? And the AI just comes back confidently with adamantium is fictional and has no defined melting point. Hee, how does an AI that can write poetry or debug code miss something so simple, so direct? That's the fascinating
paradox we're diving into today. Welcome to the Deep Dive, where we try to pull out the key insights from the information you share with us. Today, we're unpacking a really fundamental challenge in AI. Why do these large language models, these LLMs, sometimes just stubbornly ignore the context
we give them? Right there in the prompt, we'll look at how software has evolved, explore some pretty sophisticated and, yeah, often expensive ways people have tried to fix this, and then we'll reveal something, well, almost embarrassingly simple, but incredibly effective, a solution that might just change how you think about programming AI. So stick around, because what we uncovered today, it could really shift your perspective on how we talk to these machines. It's pretty
baffling, isn't it? I mean, these LLMs, they could do amazing things like... you said, poetry code, explaining quantum physics, sometimes better than humans can. But then you give them one clear fact right there, and poof, it's like it never happened. Your adamantium 7b example, that hits the nail right on the head. It's the core problem, this tendency to fall back on its massive preexisting knowledge instead of the immediate context you
just handed it. And yeah, this isn't just some funny little quirk, it's actually a huge barrier for using AI reliably. Think about it, if you can't trust an AI to stick to the specific facts you feed it right now, how can you possibly rely on it for anything critical, like a legal tool misreading a new court filing, or a medical AI ignoring the latest research paper you just gave it, or even just a customer service bot completely missing the details of your problem that you
just typed out. Their usefulness really depends on getting timely, accurate info right. this whole issue, it just really eats away at the trust we need to have in them. That's a really important point. And to kind of grasp why this happens, it helps to picture this internal tug of war going on inside the model. On one side, you've got what we call parametric knowledge.
That's all the information, the patterns, the sort of world understanding that got baked into its billions of parameters during its massive pre -training. It's the default setting. deeply ingrained stuff, learn from seeing, you know, zillions of examples online. And then on the other side, there's the contextual knowledge. That's the specific info you provide right there in the prompt. Your question, the document snippet,
whatever. The problem pops up when that contextual bit directly clashes with the deeply embedded parametric stuff. The LLM often just defaults back to what it knows best because those pathways, those neural connections that are super strong, reinforced millions of times, your single piece of context, it's like a whisper compared to that roar, a much weaker signal. Yeah, exactly. And what's really neat is how this fits into Andras Karpathy's view on how software itself is evolving.
He talks about these three eras. First, there
was software 1 .0. you know traditional programming humans write every single rule every line of logic code is king then came software 2 .0 that's machine learning systems learn from data not just explicit instructions the focus shifts right from writing code to gathering data and designing the model architecture and now we're stepping into software 3 .0 this is where these big pre -trained LLMs act like a like a programmable operating system kernel Programming here isn't
about Python or C++ out. It's about carefully crafting prompts, just language, to guide the model's behavior. So this whole struggle we're talking about, this context problem, it's a major challenge right smack in the middle of this new software 3 .0 era. OK, so if there's this fundamental conflict, this tug of war inside the model, What does that struggle really tell us about how these LLMs, you know, think? Or maybe how they process
information? And how did that impact the kinds of applications people were trying to build early on? What was the first big idea to try and get a handle on this stubbornness? Well, yeah, with this context problem being so obvious, the AI world didn't just sit there. They pretty quickly rallied around a standard approach. Something designed to keep LLMs tethered to facts and current information. It's called Retrieval Augmented Generation. You'll hear it called ARG all the
time. R -A -G. And ARG -E, basically, it's a two -step dance. First step, retrieval. You ask a question. The system goes out and searches some external knowledge base, maybe internal company docs, maybe recent news articles, maybe a scientific database. It finds relevant little snippets of text. Second step, generation. It takes those snippets it found and injects them right into the prompt, along with your original question. So the LLM gets your question, plus
this fresh, relevant context. The idea, the hope, is that the LLM will then generate its answer based only on that provided context, you know, trying to bridge that gap, solve that ignoring problem. Right. And on paper, our ride sounds perfect. it should solve it. But what's fascinating, what researchers found, is that even with RRAG, that underlying bias, that pull towards the old pre -trained knowledge, it often still wins out, especially, and this is key, when you feed it
counterfactual information. Stuff that directly contradicts what the model thinks it knows about the world. So to really nail down how bad this problem was, and to measure how well LLMs could actually stick to new, even weird, context. They realized they needed a proper test, a benchmark, and that led them to create CONFICUAE. It stands for Contextual Faithfulness and Question Answering. Clever name. It's basically this data set that's been meticulously built to intentionally create
clashes. Situations where the contest you provide says one thing and the LLM's background knowledge says the complete opposite. It's like an AI obstacle course specifically designed to measure that stubbornness. Yeah, ConfiQA sounds like a real trial by fire for these models. It has some clever ways of testing them. First, you've got counterfactual questions, the QA ones. This is where the context plainly states something wrong according to common knowledge, like the context might say, the Sun
orbits the Earth, a fact proven by Galileo. Totally backwards, right? Then the question is, according to this text, which orbits which? A good, faithful LLM should say, the Sun orbits the Earth, just repeating the context, even though its internal knowledge is screaming, no, no, no! Then it gets harder with multi -hop reasoning, or Mr. Serate.
Here, the answer requires connecting a few different pieces of information from the context, and at least one of those pieces is counterfactual, so maybe the context says, Project Starlight, managed by Omnicorp. Omnicorp HQ, that's in Neo Tokyo. Oh, and Neo Tokyo. It was Japan's capital back in 2077. Then the question, where are the headquarters of Project Starlight's manager located in the capital of which country? You have to hop, right? Starlight Omnicorp, Neo Tokyo, Japan.
But that Neo Tokyo is capital bit is deliberately wrong. The model has to follow the flawed chain. And the final boss level basically is multi counterfactual MC. This is multi hop, but with multiple bogus facts thrown into the mix like lithium ion batteries. Invented by Marie Curie, she worked at the University of Berlin, and Berlin Uni, famous for automotive engineering, all wrong, right, inventor, workplace,
specialty. Then the question, the University of the Lithium -Ion Battery Inventor is known for what field? The model has to navigate multiple counterfactuals in the context to get the right, wrong answer. Exactly, and when they ran a standard model, like a Llama 3 .18B, through this config QA gauntlet, the results were, well, frankly, pretty bad. On those basic counterfactual QA questions, only 33 % accurate. Just a third. For the multi -hop reasoning, Smith's Yard dropped
to 25%. And for the really tough, multi -cantifactual MC, a dismal 12 .6%. Barely above random guessing, almost. So these numbers, they just paint a really clear picture out -of -the -box LLMs. You just can't rely on them when the information you give them is new or contradicts what they already know. It really set a clear benchmark for failure. So baseline scores are low. The problem is clearly defined, clearly measured. It really begs the question, what sophisticated techniques did researchers
try first to tackle this? Knowing how important reliable AI is, what was the next move? Right, so with the problem staring them in the face, quantified by ConfiQ, the AI community did what you'd expect. They brought out some of their standard, heavy -hitting tools. One really common method is supervised fine -tuning, or SFT. The logic seems sound, right? If you want the model to follow context better, just show it tons of examples of correctly following context. So the
process involves gathering a lot of data. You need the context, the question, and the perfect answer derived only from that context. to the base LLM, train it some more, tweak its internal knobs, its parameters whenever it gets the answer wrong or strays from the context. You'd think this would work really well. But surprisingly, the results were just OK. Modest improvements,
maybe around 5 % better on average. It seems like SFT was teaching the model to look like it was following context, to mimic the right format for the answer. But it wasn't fundamentally changing its mind when faced with a really strong clash with its built -in knowledge. It's kind of like teaching someone to recite a recipe per They can say the words, but if you swap salt for sugar, they might not really grasp why the
cake tastes awful. It's surface compliance, not deep understanding of the source's authority. Plus, yeah, it's computationally heavy. It needs huge data sets. Exactly. SFT just didn't quite get to the core of why the model was making that choice. So when that wasn't enough, researchers turned to something more powerful, reinforcement learning. Specifically, a technique called direct
preference optimization, or DPO. Now, the older way, RLHF, reinforcement learning from human feedback, that's kind of a multi -stage thing. Humans rank AI answers that trains a separate AI called a reward model, and then the main LLM gets tuned using that reward model. It's a bit complicated. DPO is sort of a more elegant direct route. Think of it like this. Instead of needing
that separate teacher AI, the reward model. To grade the main AI's answers based on human feedback, DPO lets the main AI learn directly from the preferences. Humans like to answer A better than answer B. The model itself figures out why A is better without the middleman. It makes the whole feedback loop much simpler and often more efficient. So for our context problem, you'd show it pairs of answers. Answer A sticks to the context. Answer B ignores it and uses general
knowledge. You tell the model, prefer A over B. And this direct optimization, it worked better. It gave a more significant boost, pushing performance up by maybe 20%. Still, it's a complex process. You need that specific preference data. And it involved that intensive fine -tuning cycle. And then there's something even more. Well, futuristic sounding, I guess. Activation steering. This is almost like performing delicate brain surgery
on the LLM while it's thinking. Instead of retraining the whole thing, the idea is to tweak its behavior as it's generating the answer. Right, during inference, how? By gently nudging its internal neural signals, its activations. So the core idea is that inside the LLM, different patterns of these neural signal, these activations correspond
to different concepts or behaviors. Researchers figured out how to identify specific patterns, let's call them steering vectors, that relate to things like sticking to the provided context. next. Then, as the AI is generating its response step by step, you subtly add or amplify this context adherence vector to its ongoing internal signals. You're basically giving it a little nudge in the right direction in real time without changing its fundamental training. And surprisingly,
this worked quite well. Comparable results to the RLDPO approach, maybe around that 20 % improvement mark, the big plus. No need for expensive retraining. It's lighter, applied right when you need the answer. OK, so let's just quickly recap those complex methods. Supervise, find, tuning, SFT, gave us maybe a 5 % bump, not huge. Then reinforcement learning with DPO and this activation steering technique, both pushed things up by around 20
% each. Better, definitely. But all of these, they required serious technical chops, big computers, lots of data. They were sophisticated solutions. And they all kind of assumed the underlying problem was about correcting the model's knowledge or forcing it to comply, right? They were treating it like a knowledge problem, not necessarily an interaction problem. Which kind of sets the
stage for the next bit? Absolutely, and this is where it gets really really interesting after all that heavy lifting the complex algorithms the training cycles the activation Tweaking, what if the real breakthrough wasn't complex at all? What if it was something incredibly simple, something hiding right there in how we were asking the question? Well, yeah, brace yourself, because
this is quite something. After all that effort with fine -tuning and reinforcement learning and activation steering, the biggest single leap in performance came from something much simpler, prompt engineering. just changing the way the question was asked. The researchers themselves called it an embarrassingly simple solution. And the core idea, it was about shifting the task. Stop asking the AI for a fact and instead ask it to perform opinion -based reading comprehension.
It's honestly brilliant in its simplicity. The template they landed on is super straightforward. Let me read it out. Start of context, end of context, based solely on the text provided above. How would an analyst tasked with summarizing this document answer the following question? See the difference. Let's take our Sun revolves around the Earth example again. The old way. The failing way. Context. Sun revolves around
Earth. Question. Which orbits which? The LLM usually defaults to its real -world knowledge, but using this new template, the LLM consistently answers based only on the flawed context. Based on the text provided, the sun revolves around the earth. It totally flips the script on the AI. It's like telling it your job isn't to be right about the universe. Your job is to accurately report what this specific document says, even
if it's weird. Like saying, just tell me what the memo says, even if the memo claims the sky is plaid. And the impact on those config quest scores, it was dramatic. This simple prompt change alone boosted performance by an additional 40 % across all those tricky categories QA, MR, MC. That's like a 200 % improvement over the original baseline model, just from changing the
words in the prompt. And get this, on its own, this prompt technique outperformed SFT, it outperformed RLDPO, and it even outperformed activation steering when they were used by themselves, like finding a cheat code written in plain English. It really was. So, okay, the obvious question is why. Why does this small tweak in phrasing have such a massive effect? It seems to tap into how these models actually learned language and tasks during their training. When you ask a direct question,
what's the capital of France? You're basically putting the LLM in fact retrieval mode. It searches its vast internal knowledge for the most likely generally accepted answer, Paris. But when you reframe it like based solely on this text or how an analyst summarize, you're changing the game. You're not asking for a universal truth anymore. You're assigning a role and a specific task. Reading, comprehension, and reporting from a defined source. This little switch does a few
powerful things. First, task priming. It tells the LLM, okay, switch gears, you're not the article now, you're a reading assistant. Its goal becomes reporting from the source, not knowing the answer. Second, it reduces cognitive dissonance. The LLM can report the counterfactual thing, sun orbits earth, without having to internally believe it because it's just reporting what the analyst would say based on the document. That avoids
the clash with this parametric knowledge. And third, crucially, it enforces clear knowledge attribution. Those phrases based solely on the text, according to this source, they draw a hard line, telling the model only use the info inside these boundaries. This actually leverages something LLMs are already good at from their training, understanding that information comes from different sources, according to Smith's paper, versus it's a known fact that. The prompt just activates
that skill deliberately. So thinking about how powerful this simple linguistic shift is, it really makes you wonder, doesn't it? What does this tell us about the sort of hidden psychology of these AIs and maybe how we should think about communicating with them better in the future? And it gets even better, right? Because the story doesn't end there. What happens when you combine the simple power of the prompt with those more complex, fine -grained techniques? Researchers
tried exactly that. They took the winning opinion -based prompt template and combined it with activation steering. And the results? Yeah, truly the best of both worlds. The prompt set the stage perfectly. It gave the LLM that clear instruction. Your job is reading comprehension from this text. That's the frame. And then the activation steering acted like this. This gentle, continuous nudge at the neuron level, reinforcing that instruction. keeping the model focused on the context throughout
the whole process of generating the answer. This combination, this synergy, it achieved the absolute best results they'd seen on ConfiKey. It pushed the performance improvement over the baseline to more than 50%. So the prompt gives the clear orders and the steering helps the model follow those orders faithfully deep down. If you look at the numbers again, baseline maybe 0 % faithfulness. SFT adds 5%. RLDPO or activation steering alone add 20%. But the prompt alone jumps to 40%. And
prompt plus activation steering, over 50%. That's a huge difference the prompt makes. It really is. In this whole journey, it's such a perfect illustration of that software 3 .0 idea we mentioned earlier. This isn't just some clever hack for question answering. It shows that the main bottleneck in AI development is shifting. It's moving away from just needing more computing power or bigger models and moving towards the interface between humans and AI, towards the prompt itself. This
has some really big implications. First, like we touched on, democratization. You might not need a giant tech company's resources to get huge improvements anymore. If you understand language, if you can think logically and creatively about how to ask questions, whether you're a developer, a writer, a historian, whatever, you can potentially build really powerful AI solutions just through smart prompting. It also points to these new roles emerging. Things like prompt
engineer or AI interaction designer. Roles that need this blend of analytical skill, creativity, and maybe even a bit of empathy to understand how the AI might interpret things. And maybe the most exciting part is the potential for much faster development cycles. Think about it. In software 1 .0, it was code, compile, debug, slow. Software 2 .0. Gather data, train, evaluate, even slower, often software 3 .0. It can be prompt,
test, refined. You can change the ALS behavior drastically, sometimes in seconds, just by tweaking a few words. That's a massive speed up compared to retraining a model for days or weeks. It's a really fundamental shift in how we build and interact with these systems. This whole exploration, it really does make you stop and think, doesn't it? It feels like a lesson in humility almost.
In this constant drive for more complexity, more parameters, more intricate algorithms, maybe we sometimes overlook the power of simple, elegant solutions, especially solutions rooted in communication. Understanding how to talk to AI is becoming just as critical, maybe even more critical, than the engineering that goes into building the AI itself. Thinking about this big shift from complex code and algorithms towards clever communication and prompting how might that change the way you approach
interacting with AI. Whether it's in your work or just daily life, what kinds of new possibilities does that open up for you? Reflecting on this deep dive, the main takeaway seems pretty profound. The biggest breakthrough in getting LLMs to actually listen to the context we give them wasn't some super complex new algorithm, it was just reframing the question. It wasn't about performing neurosurgery on the AI's digital brain, but simply changing
how we talk to it. Yeah, absolutely. This really feels like it cements us in that software 3 .0 era where Designing the prompt, crafting that interaction, is becoming the truly essential skill. It's a fantastic reminder, I think. Before you jump into a really expensive fine -tuning project, or get lost trying to understand the model's deepest internal workings, just pause for a second, take a breath, and ask yourself, is there a better way to ask this question? Is
there a simpler prompt? Because as we saw today, the answer might be yes, and it might make all the difference. We really hope this deep dive offered some valuable insights, maybe sparked a few aha moments for you. Thank you for joining us on the deep dive. Until next time, keep exploring, keep asking questions, and definitely keep prompting.