Multimodal AI: How Machines Are Learning to See, Hear, and Reason

Welcome to the Sentient Code, where intelligence is engineered, autonomy is emerging, and a line between human and machine grows thinner. Each episode, we decode the algorithms, explore the robotics, and examine the ideas shaping the future of artificial minds.

I want to start today by asking you to do something that feels incredibly simple, almost you know, trivial, but it's actually a miracle of biology. Right now, just pause for a second and notice exactly what you're doing. You're listening to my voice, obviously, but maybe you're also driving, so your eyes are scanning the road, watching for break lights. You feel the texture of the steering wheel under your hands. Maybe you're drinking coffee and you can smell the roast.

It's the sensory soup. We're swimming in it.

Exactly, it's a soup. But here's the thing, and I really want you to catch this. You aren't toggling between these senses like you're switching apps on a phone. No, you don't stop hearing to start seeing. You don't pause your sense of smell to process the texture of the wheel. Your brain is this incredible fluid mixing board. It takes audio visual tactile and textual inputs and weaves them into this single, seamless narrative we call reality.

And it's completely effortless for us. I mean, it is the defining feature of biological consciousness. So we don't really think about modalities, do we. We just think about the world.

But and this is the big concept we're unpacking today. Until very very recently, artificial intelligence was not like that at all. In fact, it was the exact opposite.

Oh, it was completely fragmented. You look at the history of AI really from the nineteen fifties up until well the early twenty twenties, we were building a fractured mind. We had what we call the island problem.

The island problem. I like that image paid the picture for us.

Okay, so picture an archipelago on one island. You have these brilliant computer vision systems. They were specialists. They could look at a photo of a cat and tell you that's a tabby with like ninety nine percent accuracy. Superhuman vision in some respects. Right. But if you showed that same system a handwritten note that said this is a cat, it was blind. It couldn't read. It had no concept of what letters were.

Okay, so that's island one the eye they cannot read exactly.

Then on the next island over you have the text spots. The ancestors of you know, chatchypt and the like. They could write you a sonnet about a cat. They could define the biology of a feline. They could translate cat into fifty languages. But if you showed them a picture of a kitten, nothing, just static. They were effectively brains in a jar that only knew the world through symbols.

So you have the eye that cannot read and the brain that cannot see precisely.

And the worst part, they were built by different people. The computer vision engineers didn't hang out with the natural language processing engineers.

They were in different departments.

They used different math, they used different architectures. They were effectively different species of intelligence.

So for fifty sixty years we were building these savants. One savann could see perfect pixels, one savant could pars perfect grammar. But they couldn't have a conversation.

They couldn't even acknowledge each other's existence.

And today, because the reason we're doing this show is that something fundamental changed.

Today. The bridges have been built the water between the Islands is gone. We are witnessing the rise of multimodal AI and I want to be really clear to everyone listening, this isn't just a feature update. This isn't just now your chatbot has a camera icon.

It feels much much bigger than that.

It is fundamental. We are moving from the era of the specialist to the era of the generalist. We are giving machines the ability to integrate senses in a way that well, it mimics that human sensory soup we started with.

That's our mission for this discussion. We've pulled together a stack of research, technical papers, and industry analysis to figure out how this happened, because it seemed like for decades we were stuck and then in the last few years everything just collided right.

We're going to look at the architecture, the actual aha moment that let machines see and read. At the same time, we'll look at the superpowers this unlocks, like reading X rays while reading patient notes.

And we absolutely have to talk about the limitations because the research shows that all these machines can see, they hallucinate in brand new.

Ways they do, and we need to ask the big philosophical question if a machine looks at a photo of a funeral and writes a palm that makes you cry, does it actually understand grief or is it just really really good at math.

Let's get into the mechanics then, Section one. How did we get here? Because I remember reading about AI in say twenty fifteen, and it was all about these specialized tools. You had one tool for chess, one tool for translating French. When did the walls come down? Was there like a single invention.

To understand the solution, you really have to understand why the walls were there in the first place. And it all boils down to the architecture, the literal shape of the neural networks.

Okay, break that down for us, you know, don't go too heavy on the jargon, but give us the reality. Why couldn't vision bot talk to the text butt?

Okay? So, for a long time, the king of computer vision was something called a CNN, a convolutional neural network.

We've touched on these before. These are the ones that scan an image like a grid, right, looking for edges and shapes. Right.

Imagine a sliding window moving over a picture. It looks at a tiny patch of pixel, say a three x three square and asks is there an edge?

Here?

Is there a curve? Is there a color gradient? It builds up from lines to shapes, to ears to eventually a cat. It is designed mathematically to process grids of spatial data. It understands space.

Okay, so that's the eye. It deals in grids. It thinks and grids exactly.

But for text, text is in a grid. Text is a stream, It's a sequence the quickbat brown dot fox. The order matters immensely. Of course, you can't just look at brown without knowing quick came before it. So for that we used Ours were current neural networks. These were designed to remember the past. They process the word fox while trying to hold onto the memory of the.

So you have one kind of math designed for grids, which is space, and a completely different kind of math designed for streams, which is time.

You've got it. And you couldn't just plug one into the other. They spoke different languages. It was like trying to put a VHS tape into a toaster. The inputs just didn't match the machinery.

So what changed? I know the answer involves transformers, because that seems to be the answer to everything in AI. Lately. But why what did the transformer do that the others couldn't.

Well, the date was twenty seventeen. The paper was attention is all you need. We talk about it all the time on this show. But the hidden revolution in that paper wasn't just that it was better at language. It was that the transformer was a universal substrate.

Universal substrate, Yeah, that sounds impressive, but what does it actually mean? In practice?

It means it's a structure that can process any kind of information as long as you can turn that information into a sequence.

So text is obviously a sequence, word after word after word that fits right.

In AI terms, we call those tokens. But then the researchers have this, this real aha moment. They realized, wait a minute, we can treat an image as a sequence too.

Hold on, how do you turn a picture into a sequence. A picture is a flat two D object. It doesn't have a start and end like a sentence does.

That was the stroke of genius. The researchers asked, what if we forced it to be a sequence?

Forced it? How?

Imagine taking a photo of a dog. Now imagine taking a pair of scissors and cutting it up into a grid of little squares. Let's say sixteen by sixteen pixel squares. You have a pile of these tiny patches.

Okay, I'm with you.

Now, you just line them up square one, score two, scure three from top left to bottom right.

You flatten the grid into a line exactly.

They turn the image into a sentence of visual words. They call them patches. And once you did that, once you turn the image into a sequence of patches, the transformer looked at it and said, I know what to do with this.

Because of the transformer, a patch of pixels is just another token, the same way a word is a token.

Precisely, that is, the everything is a token realization. And it didn't stop at images. Audio that's just a sequence of spectrogram slices. Video that's just a sequence of frames in temporal order, even code or molecules.

So the machine stops seeing image versus text versus audio, and just start seeing data stream versus data strue.

Correct. It was like discovering that French, Mandarin and mathematics are all actually dialects of the same underlying language. Once they realized that the transformer could handle all of these as sequences, the barrier between the senses just it evaporated.

That is wild. So the architecture was the lock, and this idea of tokenization was the key that fit everything.

That's a beautiful way to put it. And once that architectural problem was solved, the floodgates opened. We moved into this phase of connecting the dots, of teaching these different senses to talk to each other.

Okay, I get the architecture. That makes sense. We can now feed everything into the same kind of machine. But I'm still stuck on the understanding part. Just because I feed a picture of a dog and the word dog into the same machine, how does the machine know they refer to the same thing. Surely it's not just looking it up in a dictionary.

No, no, it's not a lookup table at all. It's geometry.

Geometry. You're gonna have to explain that one. How does a picture of a dog become geometry?

This is where we have to talk about vectors and high dimensional space, and to do that we have to talk about how these things are actually trained. The most famous example is a model called clap from open ai clip.

I've seen that mentioned contrast of language image pre training.

It's a mouthful, but the concept is really elegant. Imagine you have a massive bucket of data, and I'm talking four hundred million images scraped from the Internet and the text captions that came with them.

So like IMG zero zero one dot jpeg and the alt text that says a golden retriever catching a frisbee on the beach.

Right now, you start with a blank brain. It knows nothing. You show at the image and you show at the text. Initially, the machine thinks these are totally unrelated things. It turns the image into a set of numbers. We call that a vector, and it turns the text into another set of numbers. And those numbers are in this mathematical space, nowhere near each other. The strangers in the map total strangers. But then you apply something called contrastive loss. This is

You're forcing them to be neighbors exactly.

And simultaneously you show at the text a golden retriever catching a frisbee and a picture of a toaster, and you say, push these apart. These are not the same. These live on opposite sides of the universe.

So it's this constant game of hot and cold, pushing and pulling.

Done billions and billions of times, over and over, and eventually the machine builds a map. We call it a high dimensional vector space. Imagine a graph, but instead of two or three axes, it has thousands. In this map, the coordinates for the visual pattern of fur, floppy, ears and tail end up located at the exact same coordinates as the linguistic pattern for the word dog.

Wow. So it's not using a dictionary. It's not looking up dog equals animal. It's mapping the concept of dogness to a specific location in this massive, invisible space.

Yes, and this is why it feels like it understands because that space has geometry. It has a kind of logic.

Okay, give me an example of that logic, because logic implies it can do reasoning, not just matching.

Okay, think about the classic relationship between king and queen in text. If you take the math vector for the word king, subtract the vector for man, and then add the vector for woman, you land almost perfectly on the vector for queen.

Right. That's the famous example King minus man plus woman equals queen. It's like vector arithmetic.

Now do it with images. If you take the visual vector of a king a photo of a guy in a crown, subtract the visual features that represent man, and add the visual features that represent woman, the machine generates an image of a.

Queen that is mind blowing. The logic, the geometric relationship it holds up across the senses it does.

It means the machine has found a concept layer that sits deeper than language and deeper than pixels. It has found the meaning that connects them.

It's performing analogical reasoning.

It is. That's how the system can look at a photo of a funeral and connect it to the text A moment of grief. It's not because it memorized that specific photo and caption pair. It's because the visual information in the funeral photo and the concept of grief from the text live in the same emotional region of this mathematical space.

It's math the geometry of sadness.

In a mathematical sense. Yes, it has aligned the visual features of sadness with the linguistic features of sadness.

That explains so much about why these systems feel like they get it. They aren't just matching keywords. They are navigating a map of meaning.

And usually the architecture that runs this map, the sort of central brain, is a large language model. You have these specialized encoders. You can think of them as the eyes and ears that project all this information into the brain. The LM does the reasoning in that shared space, and then it can send information back out.

So the LLEN is the conductor of the orchestras, making sure the strings and the woodwinds are all playing from the same sheet music. Ceaseely, all right, So we have the history. The silos are gone with the science. It's a geometry of concepts. Now I want to talk about the utility, because cool math is great, but what can this actually do?

The capabilities are substantial, and I think we should start with what we can call vision language nuance, because we're not just talking about identifying objects anymore.

Right. This isn't just drawing a box around a cat and saying cat ninety nine percent confidence. That was like twenty fifteen era. AI.

No, No, Now, it's about identifying relationships emotional tenor. It can look at a scene and say this is a tense negotiation happening in a corporate boardroom based on the body language, the lighting, the arrangement of people. But one of the most practical superpowers is something called OCR integration optical character recognition.

But OCR has been round since the nineties. My scanner came with it. Why is this a big deal? Now?

Old OCR was dumb. It just scraped text off a page. It didn't know where the text was or what it meant in context. Multimodal AI reads the text in context. It can look at a street sign in a photo, read the sign, look at the cars, look at the time of day, and tell you if parking is legal right.

Now, or to go back to our earlier point. It could read a handwritten note on a medical scan and understand how that note relates to the X ray.

Itself exactly, which segues perfectly into the second big capability document understanding. This is what some people are calling the ultimate office assistant.

This is the one that I think is going to change a lot of white collar work. I want you to walk me through a scenario here, because I deal with PDFs all day and they are where data goes to die.

Okay, picture this. You have a fifty page annual report. It's got three columns of text, complex bar charts, photos with captions, footnotes. For old AI, that was a complete nightmare. The text would get jumbled, the chart was invisible.

I was soup. You'd copy paste it into a text file and just get absolute garbage. Right.

But a multimodal system sees the document like a human does. It understands the layout. It can look at the bar chart, extract the data from the visual bars, I mean literally measuring the pixels of the bars, read the surrounding text, understand what that data means, and answer a question like based on the chart on page three, which quarter had the highest revenue.

Without a human having to manually turn that chart into an Excel sheet first zero preprocessing.

It just looks and understands.

That's incredible. It basically unlocks all the information that is trapped inside images, within documents. What about audio and video You mentioned earlier that video is just a sequence of frames.

Audio and video are huge frontiers now. In audio, we aren't just transcribing speech to text anymore. We are analyzing the vocal characteristics. The system can detect emotion. Is the speaker angry, nervous, sarcastically happy?

You can hear the scare quotes in your voice.

It can absolutely and it can analyze music, not just the genre, but the rhythm, the mood, the instrumentation. When you combine that with video, you get narrative understanding. It can track events over time and start to build a story.

But the real magic, and the research we looked at was really emphatic about this is the killer app of true integration. Is it not just being good at video or good at text. It's the combo.

It is the synthesis. That's where the real power is. Let's look at a coding scenario. Imagine you're a developer. You're stuck. You get some cryptic error message. You take a screenshot of your error message. You just paste it into the AI. The AI reads the screenshot, looks at your actual code file, consults the official software documentation online, and synthesizes an answer.

So it's using its eyes and it's reading comprehension at the exact same time to solve one problem.

Or take medicine, the radiologist assistant idea. It looks that the CT scan, that's vision. It reads the patient's history notes, that's text. It checks the latest research papers for medical journals. More text, and it synthesizes a potential diagnosis based on all three modalities.

It becomes the ultimate second opinion engine.

Right or a final example in design, you sketch a rough idea for an app on a napkin, You take a photo, you upload it, and you say, make this look like a sleek modern app interface, but use our official brand colors from this attached pdf. It sees the sketch, it reads your brief, it consults the PDF for the color codes, and it generates the final image.

It's closing the loop between idea, instruction, and creation. It feels like we're getting closer to that Jarvis from Ironman Fantasy, the as system that just handles them.

We are getting closer. But and this is a very very big up. We have to talk about where it breaks because it is not Jarvis yet, and you and I need to be clear that this isn't magic. It breaks in some surprisingly dumb ways.

You don't want to play the skeptic here for a minute, because it sounds perfect, but I know it's not. Where does the machine stumble? What trips it up?

It stumbles in some surprisingly fundamental areas. The first one, and this is almost ironic is spatial reasoning, which.

Is funny, right because you'd think a computer vision system would be great at space.

It sees pixels, you would think, But remember these systems are trained on flat two D images from the Internet. They struggle to build an intuitive three D model of the world. If you show it a picture of a table with a messy pile of objects, and you ask is the apple behind the book or in front of it, it often gets really confused.

It sees the pixels of the apple and the pixels of the book, but it doesn't get the depth the physics of one object including another.

It lacks a physics engine in its head. It doesn't intuitively understand that solid objects occupy space and can't pass through each other. And this is a massive problem for robotics. If you want a robot to clean your kitchen, it needs to know exactly where the cup is relative to the edge of the table. Close enough isn't good enough. When you're hands fine china.

That makes a ton of sense. It sees the picture, but it doesn't understand the physical reality behind the picture exactly.

Then there is temporal understanding. We can process video as a stream of frames, but Understanding causality over time is really hard for these models.

Causealit you mean, like the glass broke because the ball.

Hit it exactly that. Or even following a complex argument in a lecture, if I say A happened which led to B, but then C came along and prevented D from occurring, the AI might track all the nouns but lose the thread of the logic that question what happened because of what is a surprisingly difficult cognitive task for it.

It's the difference between seeing a series of snapshots and understanding a story with a plot precisely.

And then we have to talk about the big one hallucination.

We know text models lie, they make up facts, they make up sources. Do multimodal models lie in the same.

Way, they lie in new and excitingly creative ways. We call it multimodal hallucinating.

That sounds terrifying. Give me an example of what that looks like.

It can be. A system might read the text in a financial report perfectly, but then look at a graph on the same page and completely hallucinated trend that isn't there. It might say sales are trending upwards when the line is clearly going down whoa. Or it might describe a photo and just invent details about objects that are partially hidden.

So it sees half a car behind a building and says, there is a red convertible with a dog in the back seat, even though it can't possibly see the back seat.

Right, it's confabulating based on probability things well, usually cars have things in back seats, and it just fills in the blanks with a plausible story. In a creative context, you might call that imagination. But in a medical or illegal context that's malpractice.

That's a crucial distinction. It's just guessing, and sometimes a guess is wrong with total unwavering confidence.

And there's also the problem of compositionality. This is what I like to call the catenoid at the dog problem. Explain that one identifying cat and dog in a picture is easy, that's basic object recognition, but understanding their relationship the cat is annoyed at the dog requires understanding subtle cues and the interaction between the two. Current systems often struggle to bind those attributes correctly. They might see a happy dog in an angry cat and output the sentence

They get all the ingredients right, but they get the recipe completely wrong.

A perfect analogy, and finally, the deepest and most philosophical limitation grounding, or what we can call the fire problem.

This was the part of the research that really stuck with me, the idea that the AI knows fire, but it doesn't know fire.

It's the gap between data and experience. The AI has seen a billion pixels of fire, it has read a trillion words about heat, burning, smoke, and danger, but it has never felt heat. It has never reflexively pulled its hand away from a hot stove. It lacks sensory consequence.

But does that matter. I've ever been to Mars, but I feel like I know a lot about it. I learned it all from books and pictures. If the AI tells me don't touch the fire it's dangerous, does it matter that it's never been burned itself.

That is the big counter argument. Maybe you don't need a body to understand, but there is a very strong hypothesis in cognitive science that true intelligence requires embodiment. That you can't really think about the physical world unless you have a body that risks being hurt by it. If you don't fear the fire, do you really understand danger or do you just know the statistical correlation between the token danger and the token fire.

That's deep. We should definitely circle back to that later, but first let's look at where this is hitting the ground right now, despite the hallucinations and the lack of a body, where are these systems actually transforming the world today.

Medicine is the big one. I really can't overstate this. Medicine has always always been multimodal.

Right, you go to the doctor, they look at you, they listen to your lungs, they re your chart, they look at your lab results. It's a mix of everything exactly.

A doctor is at their core an information integrator. Multimodal AI is the first tool that really matches that workflow. It acts as a high speed second opinion. It's not replacing the doctor's judgment, but it's synthesizing the data faster than any human could ever hope to.

It's the ultimate intern.

It's an intern that has read every single medical paper ever published and fields like pathology, radiology, genomics. It's starting to find patterns that humans miss. It might see a faint correlation between a genetic marker mentioned in the text of a patient's file and a specific cell shape in a microscopy image that a human would never connect because the data is just too vast.

Then there is accessibility. This feels like one of the most immediate and unambiguously positive impacts.

It is democratization on a massive scale. Think about what this fluidity between senses means. If you are blind, the world is opaque to visual signals. Multimodal AI can describe the visual world to you in text or audio.

There's a blue car approaching on your left, or the light just turned.

Green exactly, or even you are holding the can of soup upside down. It gives you eyes. For deaf users, it can translate audio to text, but also describe the emotion in the speaker's voice. It bridges the gap between the sense you have and the information you need.

It completely removes the friction of format.

Domain number three science and education. In science, we are absolutely drowning in data. We have microscopy images, protein structures, satellite data, research papers. No single human can read it all.

So the AI becomes a kind of research partner.

It becomes an active participant. It can read all the latest papers, look at all the new experimental slides, and say, hey, this pattern in the satellite data over the Amazon matches this obscure theory from a paper published in nineteen ninety. It connects dots that are separated by decades and disciplines.

And in education, how does it play out there?

Responsive tutoring imagine AI that watches a student solve a math problem on a piece of paper, literally watches the pen move through the camera and at the same time listens to them talk through their reasoning. It can pinpoint exactly where the logic broke down. It doesn't just say wrong answer, It says you forgot to carry the one in the tens column right here.

That's the difference between a textbook and a real teacher. A teacher watches the process, not just the result it is.

And finally, of course, creative work. This is the controversial one.

Text to image, text to video. We see this everywhere now it's exploded.

It has democratized visual creation in a way we've never seen. You don't need to know how to draw or paint to create the stunning image anymore. But it creates this massive tension regarding the displacement of professionals and the ethics of using copyrighted work in training.

Data artists are rightfully saying, Hey, you train this model on my entire life's work without my permission, and now it's competing with me for jobs, and it's.

A completely valid conflict. There's no easy answer. But the tenential is also there for using these tools as extensions of human vision. It's a tool that can amplify a creativity, not just replace it. A director can visualize an entire storyboard in seconds. An architect can iterate on a dozen building facades instantly.

I want to pivot back to that philosophical moment we touched on earlier, the grief example. So this is the one that really keeps me up at night.

Let's go back to it. It's the most important question, I think.

Okay, So if the machine recognizes the funeral, it correctly identifies the sadness in people's faces, and it writes a poem about loss that makes me cry, has it understood grief?

This is the critical distinction between what we call behavioral performance versus experiential understanding.

Behavior versus experience. Okay, break that down for me.

Behaviorally, yes, absolutely, it performed the task of understanding grief perfectly. It recognized the symbols, It generated the appropriate linguistic response. It passed the Turing test for sadness with flying colors. But experientially, experientially know it is a hollow shell. It processes the symbols of grief without the reference. It has the map, but it has never visited the territory. It has never lost anyone. It has never felt that hollow ache of absence in its chest.

So why does this distinction matter? I mean, if the poem is good, who cares that the poet is a sad robot or a sad human? If the output is the same, why does the internal state matter so much?

For writing a poem, maybe it doesn't matter. For generating ad copy it definitely doesn't matter. But for moral judgment, for empathy, for wisdom, that gap is critical. If we ask an AI to make decisions about elder care or legal sentencing or childcare, do we want a system that just mimics wisdom or one that actually has it?

That is a chilling thought. We shouldn't mistake a high confidence output for lived experience.

Exactly, A human radiologist brings years of seeing patients, of knowing the fear in their eyes when they get a bad diagnosis, of understanding the weight of that responsibility. The AI brings pattern matching on a massive data set. Those are not the same thing. Even if the diagnosis it gives is correct. The AI doesn't care if the patient lives or dies. It just cares about minimizing the loss function in its training.

So where does this all go next? If this is where we are now on the frontier, what's beyond the frontier?

The trajectory is becoming very clear. We are moving from processing information to acting on it.

The agentic shift. I keep hearing this term popping up more and more.

Yes, right now, most people interact with AI by chatting, write this for me, analyze this data. The next wave is agents that do things.

So not just tell me how to book a flight, but literally book me the cheapest flight to Chicago next Tuesday.

Book the flight, email my boss to let them know I'll be out, update my calendar, and order a car to take me to the airport. These are systems that will browse the web, operate software on your computer, and execute code to accomplish goals. They will have eyes to see the screen, and hands whether virtual or robotic, to click the.

Buttons, and robots real physical robots in the world.

That's the physical manifestation of the same idea. Robots that watch a human demonstrate a task, say folding laundry or assembling a circuit board, and then replicate it. The visual understanding guides the motor control. The senses are connected to the limbs.

This raises the stake significantly.

It completely changes the risk profile. A chat butt that writes a bad poem is embarrassing. A robot that misunderstands the command clean up the kitchen and throws out your vital medication is dangerous.

Or an autonomous software agent that misunderstands a financial instruction and executes a code that deletes a critical.

Database exactly when perception leads directly to physical or digital action. In the world, safety isn't just about content moderation anymore. It's about physical safety and operational security. We are giving these systems hands, we need to be very very sure about the brain that's guiding them.

It really feels like we are standing on the edge of a profoundly different world.

We are the walls between the senses are gone, the silo are broken.

So to kind of summarize our journey today, we started with the island problem. AI was completely fragmented. We moved to the universal substrate, transformers and tokens connected everything. We saw the magic of vector space, where meanings are mapped out geometrically. We looked at the superpowers, the medical assistant, the coder's buddy. We acknowledged the very real limitations no body, no real spatial sense, the problem of hallucination. And we've just looked ahead at the agentic future.

That is the arc. It's the collapse of separation.

What's the final thought here which we walk away thinking about as we go about our day.

I think it's this. The machines are learning to see and hear, They are developing senses. For decades, we have spent so much time worrying about whether they can think that we maybe haven't paid enough attention to what it means that they can perceive, perceive us, perceive the world. We are building a new kind of observer. It's not human, but it's not blind anymore. The question we all need to ask is are we paying enough attention to what that means for the world we are building.

That is a question I think we will be wrestling with for a very long time. Next time you look at your phone, just remember it might be looking back at you and it's finally starting to understand what it sees. Thanks for joining us on this exploration A pleasure, as always,