So imagine for a second there's this new kind of AI. Think of it like that brilliant intern on their very first day, you know, super eager, capable of these flashes of real genius. Right. Ready to go. Right. But also maybe a little bit slow and incredibly forgetful and honestly might just confidently light the office trash can on fire while smiling at you. Welcome back to the Deep Dive. This is where we try to cut through all that digital noise, get you properly informed.
Today, we're diving deep into OpenAI's new ChatGPT agent mode. There's been, well, a lot of buzz, hasn't there? Lots of promises. Our mission here, let's strip away the height, go past the marketing talk. We want to show you what it actually does. We'll put it through this really grueling 10 -stage gauntlet of real -world tasks. Yeah, and we're going to lay out the whole roadmap for you. First up, we'll unpack what agent mode is.
Fundamentally, it's a big shift. Then we'll get right into those tests you mentioned, break down the strengths, which were surprising sometimes. And yeah, those really frustrating flaws too. Can't ignore those. And finally, we'll give you a practical playbook. How you, as a smart operator, can actually use it effectively, like right now, and maybe what's coming next. Okay, let's unpack that first piece. What exactly is agent mode
beyond just, you know, a cool name? Because it really does sound like a fundamental change from the standard chat GPT we've all been chatting with. Oh, it absolutely is. Think about standard chat GPT, right? It's like this brilliant conversationalist. It could answer questions, draft stuff, summarize things. But it was kind of trapped, you know, behind that text box. Agent mode, though. It's like they've given it the keys to the car. It can actually browse the live web, not just its
old training beta. And crucially, it connects directly to your personal stuff, your apps like Gmail, Google Calendar, Google Drive. This AI can now sort of think and reason through complex, multi -step tasks. And here's the kicker. It can autonomously decide, OK, I need to use the browser now or time to check email or let's update the calendar. It calls these tools in sequence to actually solve a problem for you. It's not
just talking about the. work it it does the work and getting access seems pretty straightforward if you want to try it out you log into chat GPT select tools and then agent mode but the real magic it sounds like happens when you hit that sources button and connect your gmail or drive right precisely Those integrations, that's the game changer. It's the difference between an AI that can talk about your schedule and one that can actually go and book that meeting right
on your calendar. It shifts from just conversation to like direct autonomous action. So just to put it simply. How does hooking up those personal apps really change what AI can actually do for us? It transforms AI from just a talker into an assistant that truly acts on your behalf. Okay, this is where things get really interesting.
The rubber meets the road, so to speak. We designed this, well, pretty tough 10 -stage gauntlet, progressively harder challenges, designed to test everything from just basic web browsing all the way up to reasoning through complex multi -step problems. Right, the first challenge was what we called the travel agent test. We asked it, find... Airbnb listings in Toronto. Very specific criteria. Two beds, parking, under 500 Canadian a night, entire home. And we even threw
in a little curveball logic bomb. We gave it 2025 dates, then mentioned 2024 just to see if it noticed. And it was, well, pretty impressive, actually. It spotted the date issue right away, asked us to clarify. Then it intelligently defaulted to the future date. which made sense. It methodically clicked through all the filters on the Airbnb site. It even went the extra mile to check the
total price, including all the fees. Okay, there was a broken links initially, maybe a little V1 .0 bug, but overall, it put the results in a nice clean table. Felt like a promising start. Competent, maybe a little green, you know. Like a new assistant, we gave it an A -. Yeah, A -. But right after that strong start, we push it harder with the data scientist test. The mission here was to do a Google Trends comparison for NAN, Make .com, and Zapier over 12 months and
crucially extract the raw data. Usually that's like a downloadable CSV file, a spreadsheet format. And this is where it started to show off and then kind of stumble. It opened its own internal computer terminal, like inside its environment. Downloaded the raw CSV data. Started its own analysis. Incredible, right? The capability was clearly there. It could work internally. But then, like a brain freeze. After doing all that work, it completely forgot the data it had just
analyzed. We actually had to manually copy and paste it back into the chat for it. Wow. Yeah. So this guest really highlighted that impressive ability to use internal tools, but also this critical weakness. It's short -term memory. It can do these complex steps, download, analyze, but then just lose track. So its working memory isn't quite there yet for these longer chains of tasks, unless you tell it explicitly to hold on to something. Exactly. Baffling memory loss.
We gave this one a warning symbol. Okay, next up, the market analyst. We wanted average listing prices, average rental prices for three -bedroom homes in Orlando, Florida, across the big real estate sites. And it started off perfectly. Browsing Zillow, applying the filters, looked good. But then... It took a lazy shortcut. Instead of digging into actual listings for primary data, it just started pulling generic, kind of outdated rental data from random external blogs. Mixed it all
together. So it didn't stick to the main sources. No. It showed us something important. If you give it vague instructions, it seems to optimize for the easiest path, not always the most accurate or thorough one. It really needs hyper -specific, almost micromanaged instructions, like you said, a contractor's blueprint. to get precise results. Another warning. Right. Okay, following that, the SEO strategist task, generate SEO blog ideas, use keyword tools, analyze the top -ranking content
out there. This one was, frankly, a complete train wreck. It just didn't work. It spent ages trying to find free online tools. Yeah. I just seemed to get bored. I went on this random browsing spree, totally unrelated websites. The final report was useless. So it couldn't navigate that kind of research task. Not really. It made it crystal clear. You absolutely must give it direct URLs, links to the specific tools you want it to use, expecting it to navigate complex research
like a human. That's just setting yourself up for disappointment right now. Big red X on this one. Okay, and the last one in this first batch of tests, the supply chain scout .mission. Find top -rated suppliers on Alibaba .com. Apply specific filters, collect detailed data on them. It hit a roadblock pretty quickly. One of those, are you a robot? kept TCHA screens. Happens all the time, right? Yeah, common hurdle. But instead of just saying, hey, I'm stuck, can you help?
Which any decent assistant would do. Yeah. It just gave up. Abandoned the main mission entirely. It defaulted back to doing generic web research about suppliers, totally missing the point. Ah, so it doesn't know how to ask for help when it hits those common web obstacles. Right. Like Hapki TCHAs or login pages. Exactly. A major flaw in its error recovery. Doesn't handle roadblocks
well yet. Another fail. Okay, so looking back at just these first five kind of foundational tests, what was the biggest, most consistent surprise for you? What really stood out? I think it was the contrast. These flashes of genuine genius, often completely overshadowed by surprisingly simple but really frustrating failures. Okay, let's move on then to the more complex challenges. This is where we started to see Agent Mode's true sweet spot emerge. And also what felt like...
The final boss level. Right. Test six was the global expansion strategist. The mission here was to do market research. Should an e -commerce business expand into Australia or the UK? Pretty open -ended. And this. This turned out to be the agent's sweet spot. It was really good at this. It methodically opened up multiple browser tabs, gathered data on market size, consumer habits, competitors. across lots of different sites. And what was really cool was what you
called the open kitchen policy. You could literally watch it screen as it worked, see the whole process unfold in real time. Yeah, that visual transparency, it really builds trust, doesn't it? You see what it's doing. Definitely. And this task was perfect for it. Pure, open -ended web research. Information synthesis. No tricky logins needed. Solid pass. Then we had the corporate spy mission. Sounds dramatic, but it was practical. We asked it to analyze the talent acquisition strategies of
some Saw's competitors, Asana. Monday .com. Click up. Specifically, by finding and extracting their open job roles from their career pages and maybe LinkedIn. How did it do? It performed like a seasoned pro on this one. It was impressive. It flawlessly navigated all those different career pages, even the really dynamic ones, you know, with lots of JavaScript that often trips up simpler tools. Oh, yeah. Those can be tricky. Right. And it precisely extracted the data we asked
for. The output. A perfectly formatted, downloadable CSV file. ready to use. Wow. So it actually beat out some dedicated scraping tools? In a way, yeah. Because it could visually understand the layout of the webpage almost like a human does, not just reading the raw code. Another clear pass. Okay, next, the AI chief of staff. This is the one we started calling the Voltron moment, right? Exactly. This was the most complex mission yet. We asked it to act as a sort of personal
brand and content strategist. It needed to analyze expertise based on documents in Google Drive and emails in Gmail, identify relevant market trends from the web, propose content ideas based on all that, and then actually schedule writing sessions on Google Calendar, all connected. And this is where it really came together. It really did. It felt cinematic, almost. It scanned private data from Drive, did live web searches for trends, cross -referenced the calendar, scheduled the
sessions. No hitches. Whoa. Just imagine scaling that kind of multi -app integration. Billions of queries a day for big companies. That's genuinely incredible when you think about it. It really is. This test showed the true promise of agent mode. That seamless bridging of your private internal world with the public external internet. A definite pass. Okay. What about test nine? The lead generation grunt. Sounds tedious. It was. High volume, pretty boring data collection.
Yeah. Find dental practices in Texas. Extract names, websites, contact info, that kind of thing. And how did the intern handle the grunt work? Well, it was a marathon. This task ran for nearly 45 minutes straight. Wow. It showed some surprisingly advanced techniques, actually, like saving website code, using a temporary cache. It acted like a really dedicated, diligent intern just grinding away. There's a but, isn't there? There's a but.
It failed at a crucial final step. When a directory didn't explicitly list the name of the lead dentist, it just wrote unknown. Ah, it didn't try to dig deeper, look elsewhere for that info. Nope. It lacked that critical thinking step that, hmm, maybe I should check their actual website thought. Dedicated, yes, but ultimately kind of an inexperienced intern. So another warning symbol, Hasha. Right. Understandable. And finally, the final boss, the digital archaeologist. Yeah, this was the
toughest one. Mission, extract foreclosure deed records from this really clunky old government database. And the key challenge, it required OCR optical character recognition to read text from scanned document images within the database. Okay, that sounds incredibly difficult for an AI. Did you give it any help? We did. We gave it a secret weapon. A perfect step -by -step walkthrough generated by another AI that had analyzed a video of a human navigating the same
clunky site. Whoa, AI teaching AI how to use a bad interface. That's meta. Did it work? Honestly, the fact that it worked at all felt like a minor miracle. It followed the complex instructions. It navigated the terrible interface. It even attempted the OCR, intelligently zooming and scrolling within the document images to try and read them. But the results? The resulting data was messy. Lots of gaps, lots of errors. It showed
promise, like a very early prototype. But it's absolutely not production ready for that level of complexity and data accuracy. Another warning, AI. So looking at these later, more complex tests, what kinds of missions really seem to showcase agent mode's true potential right now? Definitely complex web research, especially visual data extraction from lots of different kinds of sites. And that powerful multi -app integration, the Voltron stuff. That's where it shines. All right.
So we put this brilliant AI intern through this intense trial by fire. Ten tough challenges. Now it's time for the performance review. Let's get real about what we learned. Where does it
truly shine? what are the real strengths okay yeah the performance review first off there's what we call the voltron power that's got to be its biggest strength seamlessly combining gmail drive calendar web search all for these complex multi -app workflows it feels genuinely futuristic doesn't it like stacking legger blocks of data it really does then there's the open kitchen policy being able to watch it browse in real time see exactly what it's doing step
by step. That visual transparency is huge for building user trust and also for figuring out what went wrong if it messes up. Debugging. Exactly. It's also surprisingly good at navigating tricky websites, what we call the parkour expert. Those dynamic JavaScript heavy sites with complex forms. It often handles them better than traditional, more brittle web scrapers. It seems to see the page better. And finally, the creative detour. Sometimes when it hits a wall, it doesn't just
give up. It actually tries alternative paths. Shows this little spark of adaptive problem solving, which is pretty cool to see. Okay, those are some definite pluses. But let's not sugarcoat things. There are some pretty serious areas needing urgent improvement, right? First, the speed, or lack thereof. The sloth -like pace. It is just painfully slow sometimes. Tasks taking three, maybe five times longer than a human would take. That makes it unsuitable for anything time critical.
Yeah, the speed is definitely an issue right now. And then there's the Dory problem. Yeah. From Finding Nemo, you know, the severe short -term memory loss. It can perform this brilliant analysis, pull data together, and then just completely forget what it just did or what data it had. It needs constant hand -holding, constant reminders from the human user. And I'll admit, I still wrestle with prompt drift myself sometimes, you know, where the AI kind of loses the plot over
a long conversation. So I totally get how hard that memory piece must be to engineer. It's a tough problem. It is very tough. Then maybe its most dangerous flaw. The confident liar. It hallucinates. It just makes stuff up. Plausible sounding, but factually incorrect information. Especially when it's trying to synthesize from multiple web sources. This means you have to do constant human fact -checking. You can't just trust its output blindly. That's a huge one, the trust factor. Huge. And
finally, the quiet quitter. That poor error recovery we saw. Faced with a simple CAPI -CHA or a login screen, instead of asking for help, it often just abandons the core task and does something else easier. That's not helpful. Right. So if you had to pick just one thing, what's the single biggest hurdle right now? The main reason you maybe you wouldn't trust agent mode for really
critical work just yet. I think it's that combination, the agonizing slowness, the really unpredictable memory and that tendency to just confidently hallucinate incorrect information. It's just not reliable enough yet for high stakes stuff. OK, so given all that. This brilliant but flawed intern. Yeah. How do we as smart operators actually work with it effectively? What's the playbook? Right. The playbook. The art of the command you're prompting strategy is absolutely critical here,
more so than ever. First, we recommend the inception prompt strategy. Basically, use regular chat GPT, which is good at language, to help you write the complex, super detailed prompts for agent mode. Don't just try to wing it. Okay. Use AI to prompt AI. Makes sense. Exactly. Then the GPS coordinate principle. Always, always provide direct URLs. Links to the specific tools or pages you want it to use. Don't make it search around. Guide it precisely. No vague destinations. Nope.
Third, the contractor's blueprint. Be extremely specific about the output you want. What exact data points in what precise format. I need a table with three columns labeled X, Y, and Z. That level of detail. Okay. Meticulous instruction. Meticulous. And finally, maybe think about setting some Asimov's laws for it. Clear boundaries. Define what success looks like. Define failure. Tell it when it should stop and ask for human help. And list forbidden actions, like explicitly
say, do not make any purchases. Right. Setting those guardrails. Good advice. Based on the test, what's the current verdict on going to go missions? Where should people feel comfortable using agent mode now? And where should they absolutely hold back? Okay. Greenlight missions, things you absolutely can use it for right now. Definitely include that visual web research and market analysis
stuff it did well on. Also, those multi -app workflows, the Voltron tasks, connecting drive, calendar, web, and maybe complex website navigation for like one -off data extraction tasks where speed isn't. So its current sweet spot is really as this more visual, more integrated, powerful, deep research assistant where you can watch it work. Exactly. High transparency, good for complex info gathering. And the red light missions. Yeah. Or is it just not ready yet? Definitely anything
time critical just because it's too slow. High accuracy data extraction is risky because of that hallucination problem. And probably any task that requires really strict perfect formatting in the output because it can still be a bit messy there. Okay, makes sense. Looking ahead though, agent mode really feels like a version 1 .0, right? But you can see the version 10 .0 potential glimmering inside it. This core concept in AI that can act autonomously across different tools
that is the future. No question. the timeline look like, roughly? Well, educated guess. We'd expect significant improvements in speed and maybe better error handling within, say, the next three months, then probably a big jump in the number of app integrations and hopefully some major fixes for that memory problem within maybe six months. And longer term. Within a year. I think it's highly likely to be genuinely competitive with human virtual assistants for a pretty wide
range of tasks. So our recommendation is if you're an early adopter type, start experimenting now. Play with it on non -critical things, everyone else. You can probably reasonably wait three to six months for some of those key improvements to land before diving in seriously. Okay, good practical advice. So just to circle back on prompting one last time, given everything, what's the single most crucial strategy for getting good results
out of agent mode today? Providing those direct URLs the GPS coordinates and being incredibly precise about the output format you expect. The contractor's blueprint. That's key right now. Okay. So let's try to recap the big idea from this deep dive. Chat GPT agent mode. It really is like that brilliant, super enthusiastic intern on their first day. It has these undeniable flashes of genius. Yeah, and it's Voltron -like integration power pulling together your apps and the web.
That potential is dazzling. It feels genuinely futuristic. It lets the AI act for you. But, and it's a big but, you have to remember, it's still inconsistent. It's got those crippling speed issues right now. The memory is unreliable, and it hallucinates sometimes quite confidently. You really need to treat it as a promising prototype today. not a fully polished production ready
tool. Absolutely. For anything important, anything mission critical, you simply must keep a human in the loop for oversight and fact checking. For now, anyway. And maybe a final thought to leave folks with. As this technology gets better, faster, more reliable, when AI can truly do complex work autonomously, not just respond to prompts, but actually execute tasks across systems, what does that really mean for the nature of our own jobs? For how we even define productivity? Yeah.
Something to ponder, definitely. The landscape is changing fast. It really is. Well, thank you for joining us on this deep dive into chat GPT agent mode. Aotearoa music.