¶ "Idea flows: the only business metric that matters."
Today on the data driven podcast, we have the privilege of hosting none other than Jeremy Utley. Now, Jeremy isn't just any guest. He's an academic marvel and entrepreneurial spirit rolled into 1. Hailing from the prestigious corridors of Stanford as an adjunct professor, he's the kind of chap who educates the future disruptors of Silicon Valley. He is here to tell us how to get the most out of generative AI. Now on to the show.
Hello, and welcome back to Data Driven, the podcast where we explore the emergent fields of artificial intelligence, data engineering, and data science, and all the associated technologies. With me today is Jeremy Utley, who is a, adjunct professor, venture investor, and co author of the book, Idea Flow, The Only Business Metric That Matters. Welcome to the show, Jeremy. Thanks for having me. Hey, no problem. So Stanford. That's kind of a big deal. It's a it's a special place. Yeah. I'm
just trying to not get found out. I'm sure it's, you know, similar to, the, the guy on Office Space. Right? At some point, there'll the clerical error will be revealed. You'll you'll know when they move you to the basement. Right? Exactly. Exactly. Yeah. But I've been teaching at Stanford since 2009, and I've been delighted to get to learn alongside some of the most incredible students in the world and and get to study some of those
incredible innovators in the world. So, not just I may be a professor or an adjunct professor, but I really consider myself to be a front row student in in the classroom alongside my students. Very cool. Very cool. So, what what is the most important metric? I'll start right there. Well, the most important metric we call idea flows, the only business metric that matters. And the reason that we make that bold claim is because it's the only measure of your
team's capacity to solve problems. And the the only constant in our day to day lives is problems. In in our businesses, I don't know, a single business that is facing a day without problems. And so if you think about problems as the constant, then your team's capacity to solve problems is really the most important thing that you should be measuring. And yet, nobody really even knows how to
measure it. And so we talk about idea flow as the as the way to measure a team's capacity to solve problems. Interesting. Interesting. And is this is this changing now when we have the reality of AI assisted teams? Yeah. Yeah. Absolutely. That's it's a really insightful question. Yes. It does change or sorry. It has the potential to change. And yet, what our research suggests that we've conducted over the last year or so, is that sadly, it
actually doesn't change in practice. In theory, it could change but in practice it often doesn't. What's interesting, so what what are the barriers to this? Right? Because I have some thoughts on this. I know that a number of companies have basically outright banned, use of AI tools with good intentions, right, because the privacy policies, etcetera, etcetera, but in reality, people are copying and pasting sensitive stuff anyway. So, it seems like banning something outright
doesn't always work in a number of areas. But what what are the barriers? Right? Because it it it can, like you said, but in practice, what's what's what are the blockers? Ultimately, it's it's human psychology, really, is what what's, the challenge. It turns out that our expectations of the technology, are hamstringing our ability to make use of it. Because we're approaching the technology. Most teams that we studied approach the technology as an oracle. It's almost like a search
box. It's gonna give them the best answer. Right? And that's the wrong way to approach the technology. It does feel somewhat magical when you type in, you know, an enigmatic query and get a seemingly intelligent response. I mean, that feels
magical, but the teams that do that underperform. The teams that overperform are the teams that treat generative AI not as an oracle, but as a thought partner, as as a conversation partner, and iteratively work together with the AI to discover a better answer. And the irony of that is it's not very magical, actually. It feels like work. And yet, where teams that treat AI as a conversation partner arrive is light years better than teams that treat AI as an oracle perform.
Interesting. So what are the what I think I know what you're getting at in terms of treating it like an oracle versus treating it like a conversation. Because I've seen that as I do more and more of this, I hate the term prompt engineering. I hate the strong word. I have mixed feelings about the term prompt engineering because there is no one single prompt to rule them all. Mhmm. At least that's been my experience where you kind of you kind
of it's like a conversation, like, you're having. It's not a person I know. It's not a person I know. It's not a But it's but but it is a mindset, Frank. That's the thing. It's a mindset. And people don't come with the mindset of I want to have a conversation. People are lazy. Right? So so Herbert Simon, back in 1954, won the Nobel prize for what he deemed satisficing, which was the human tendency to settle for good enough. Right? And in most
of our lives, it's fine. I need a good enough pair of jeans, I need a good enough cup of coffee, whatever it is. Right? But when we're trying to
¶ Striving for the best is essential.
solve problems, good enough sometimes is is okay. But often, especially when it pertains to innovation, you don't just want the good enough thing. You want the best thing. And it's in that area where when we really want the best solution, that our tendency to settle for good enough really hurts us. Because what teams do is they put in a prompt and they get a pretty good answer and they go, woah, I I was prepared to take an hour working on this but we kind of
got pretty good in 5 minutes. You guys want to go get coffee? And everybody just gives up because they got good enough. And so that's, you know, I I think it really is a mindset thing. Forget the word prompt engineering. It's all it's it's self engineering. It's human engineering. And one of the best things that a human being can do is say to the AI why they don't like the answer the AI gave. Right? So take your expertise.
Here, this is something that anyone is listening can do right now. Take something that you know you're an expert on. So for example, I'm an expert on customer insights or or low resolution prototyping and experimentation. Right? So I might say to the AI, can you give me a step by step guide for how to conduct an experiment? K. If I
¶ Expert hesitations on using AI for knowledge.
did that, we can do it right now live if we wanted to. But if I do that, it's gonna give me like, you know, the average of the Internet. Right? And it's gonna draw from a bunch of stuff that may be good, may be bad. By the way, I'm an expert, so the chances of my knowledge surpass it are reasonable, you know, at least. But because it's gonna give me the average of the Internet, it's probably gonna find some corners that I don't know
about and it's probably gonna say some stuff that I vehemently disagree with. Well, where most people give up and I think because they want AI to not be that good is they look at the response and they go, see. It didn't even know that you're supposed to test for desirability and not feasibility. Right? Whatever. And then they say, that's why AI is no good. Well, Human Engineering, not Prompt Engineering.
Human Engineering is to say, okay, human, tell the AI what you disagree with and why you disagree with it, and ask the AI to regenerate an answer given the following considerations and put in your critique. Most people if they do that, just even that one thought exercise, will be blown away. Yeah. You get an order of magnitude better response. Absolutely. Right? Absolutely. Because you're kind of focusing the cone of, you know, inquiry with your own
expertise. And what people want is they I mean, no one would ever if you think about, like, AI like an MBA intern. Right? No one gets an intern from Harvard Business School, gives them 2 sentences of instruction and then at the end of the summer says, man, their workout was no good. I didn't interact with them at all. I didn't give them any guidance. But for crying out loud, what's
Harvard doing these days? Right? No. Nobody gives a human being 2 sentences of input and then critiques how bad of a job they did, right? And yet we open chat gpt, we give 2 sentences of input, if that by the way, and then we go, See, it's not very good, well, work with it. Garbage in garbage out. It's as old as possible. And the reason most people don't I think most people don't wanna work with it is because they don't want it to be any good. Yeah. I could see that. Totally.
And the people who do want it to be good will be unlocked and unleashed. But it requires not prompt engineering, but human copilot engineering. I do like the fact that a lot of these tools that are coming out are being called copilots, right? Because I think it shifts the focus away from AI isn't going to do it all. AI is not probably gonna take your job, right? But it's just an assistant. Right? It's it's to help you out where you may
want a little bit of boost. I also think that I think what you you described is good enough factor is I think people see large language models and they they assume it's a search box only better. Yes. Well, and part of our challenge, you know, I was talking with a with a psychologist, David McCraney, who wrote How Minds Change. He's a he's a journalist, an author, a podcaster. He's a he's the host of You Are Not So Smart, which is
all, you know, obsessed with cognitive bias, which I love. Mhmm. I love that podcast. And one of the things that David and I were talking about yeah. I can't remember the the name for the cognitive bias, but when we see something that we think we understand, we just track into our kind of typical neuro pathways. Right? So we see a text box and we go, oh, I've seen one of these before. This is like that. And we so and this this being generative AI is not like that. That being
search. Generative AI is not search. But because of the kind of the the UI, we approach it like search. We go, okay, I want the answer. Just give me a list of links that I'm gonna click through and decide on. And we don't interrogate Google, we don't critique Google or any search engine, right? We don't say why we want it
or state our intention, right? But if you start to do some of these fundamental kind of human conversational tactics, if you start treating it more like a person than like a search box, you get exponentially better results. But you're right, even the UI itself predisposes us to treat the technology and to think about the technology in a particular way and that is actually holding us back. Interesting. I noticed
this in a completely random thing. I was getting, I was using DALL E to generate images, This is before Chat GPG had it in there. And I wanted to make a painting that looked like a Rembrandt painted a portrait of a dachshund. I know this is the most ridiculous thing. Right? So I wrote the prompt, I said, you know, painting of a dachshund in style of Rembrandt, and it produced something. It was okay. Right? It was good. But I was like, I wonder what if I asked
¶ Asking for help, ended up with an image.
ChatCpt to help me with this prompt? So I went over. Now I could do it all in 1 window. But I said, like, hey. What would what would you write for a prompt? Like, what would do that? And it came back with, I mean, a paragraph to what you said, 2 sentences. This thing came back with a paragraph. I mean, stuff that only art historians and art,
students would really appreciate. You know, this type of paint, this style of brush, like, just stuff that I remember from art history class, but, like, you know, I only took that class because I had to type thing, you know? But but then I I pasted that prompt in there, and, oh, okay. It's it's it's an image. It's art. It it it's somewhat subjective, but the the result was so much better. Like, it was just day and night, and That's true. That has changed the
way I think about, dare I say, prompt engineering. Right? Like, because you can because I gave a talk on prompt engineering and, like, you know, the magic of it, and I was like, you can actually have the models help you build out prompts. Yes. Well, that's that's the thing that people don't understand is, you know, I mean, I I interviewed the other day on a I've got a podcast called Beyond the Prompt, which is all about AI in organizations. And we've interviewed a bunch of amazing people.
You have co founder of Typeform, CEO of Section, CEO of Every, the head architect at Instacart, a bunch of interesting people. And one of the folks we interviewed last week is a documentary filmmaker named Juan Carlos. And Juan Carlos has made some amazing documentaries. And he said he's always wanted to build an Ios application, but he's never had a developer and he's always seen that as
kind of prohibited. He can't do it. And then he said when ChatGPT came out, he had the thought, could ChatGPT teach me how to code? And he built an Ios app by treating ChattGPT like his computer science TA. And he would go to the TA and ask for instructions. He got Chad GPT to teach him how to build an Ios app. Nice. You would never imagine doing that with a search engine, right? No. You would find it on. You would find it on. But you would just, and anytime he got stuck, You'd
come back to the TA. Right? And you get more. But your point about people's minds being open, I think they have to be hearing examples like this. He literally went to JIGBT and said I would love to build an Ios app but I've never built anything. I don't have the first you know, sentence of ways to even describe it. If you were gonna ask a developer or if I wanted to ask a developer to do this, how would I even ask them? What do I need to describe? Tell me everything you need from me in
order to tell me how to proceed. And he basically worked with it's almost reversed. We're used to being in the driver's seat. He basically told Chad GVT, you're in the driver's seat, please tell me what to do. I'll be your hands, you tell me what I need to do. And to me, that's just we have to start shifting paradigms. I'll give you another example. I've got a good friend who is considering a job transition. He lives on the East Coast, wants to move back to where his family is, and he got a
job offer at a new firm. And he felt the job offer wasn't a great offer. His wife felt, we don't wanna screw this up. We wanna give back to family and we got a job. Just take
¶ Convince family to take job offer role-play.
the offer. And he he kind of confided in me, I feel like I could negotiate, but I don't want to mess things up. And I said, well, have you role played it with Chad GPT? And he said, what do you mean? I said, well, you can role play the conversation just to see how it would go. He said, but they don't know anything about the
firm. I said, well, you can tell them. Ask ChadGpt, what do you need to know about the firm and what do you need to know about the hiring manager in order to believably play their role in a back and forth role play with with me. Interview me about the company and interview me about the person until you know enough to believably play their role and then do a 1 on 1 negotiation with me.
Be observing the negotiation the whole time, and give me feedback not only as my counterparty, but also as a negotiation coach. That's brilliant. That's some sci fi stuff right there. Dude, he came back and he was like, what do I do now? That was mind blowing. I said, now, ask him to play your counterparty, but be a little bit more aggressive as the counterparty, a little bit less friendly. So he did that and he said, Jeremy, 2
things I learned. 1, or actually 3 things. 1, I was missing my key point of leverage and ChatGPT helped me see it. 2, I forgot my negotiating strategy in the in the heat of the moment, and chat g p t alerted me to that. Now I'm prepared. 3, I'm no longer dreading this negotiation. I know I can do it.
¶ Learning new skills and fostering imagination through technology.
Wow. And to me, it's like that's it's it's so different than saying, you know, portrait of a dash hound and, in remember it's like people are doing that and going, that's all I can do is like, you know, it can teach you how to build an Ios app. It taught me, I got you at GBT to teach me how to code Python so I could build my own chatbot using Python. I've literally never written a line of code in my entire life, right? It's our imaginations are the
primary bottleneck here. And and part of the reason that our imagination is constrained is because we've been trained by search to interact with technology in a particular way. And what I think most people need is they need to hearing examples like this and they need to be getting in conversations with other people who are trying stuff and going, I can do that. Yeah,
you could. I could do that. Yeah, you could. And you need to be having these kinds of conversations to stimulate your own thinking to then discover your own novel applications. No. That's brilliant. I mean, the whole negotiation thing is amazing. I've seen a lot of chatter online about people using it to, you know, in the job search aspect of it. Right? Like, here's the job description. Here's my current resume. Have at it, you know? Write Reno, write a cover letter that is
gonna hit all these points and it'll do it. And, you know, but I mean, the whole idea of role playing. I mean, that's just brilliant. Like, I think I think the the the the the the $1,000,000 statement there is our imagination is a limit, which is something that historically, when it comes to computers, I would say beyond the the the the the search interaction experience, we're not used to computers outthinking us. Yeah. Yeah. And I think that that that's gonna have some interesting, societal
consequences. Right? Because I mean, I think what what freaked people out about Chat GPT was, you know, it looks like it's doing something creative, which is something that we had naively assumed, was something only humans can do. Mhmm. And I I I think you're right. I mean, I think this is not just a chat search only better, but this is definitely like a whole new type of computing. Yeah. I
think it really does require a behavior modification. And what I I there there are kind of 2 big questions in my mind for organizations or for leaders who are thinking about deploying these technologies. 1 is, what percentage of my workforce is comfortable with these tools? And by the way, right now, I mean, sentiment I read an Ernst and Young report that says 70% of people are afraid
of AI. You know, it's like, when the when the predominant sentiment is fear, you're not in a position of kind of maximizing opportunity. Right? So Right. You so fear is gonna hold you back from that sense of comfort, confidence, etcetera. But then 2, so if you say so 1 question is, what percent of our workforce is comfortable? And then 2, how do I grow my conversation abilities? Nobody knows how to have a conversation right now with with HHPT or with
any LLM. Many people have lost the art of having conversations with human beings, right? So, but you you really have to almost it's like becoming literate in a new language. We need AI literacy courses. We've actually developed, my partners and I, developed a conversational coach who gives daily drills that send you into ChatTPT with kind of a drill to build your conversational fluency. Because what we're finding
is, folks just they don't have any imagination. Do you know that you could take ChatGPT, for example, and tell her what are your 5 favorite books and why they're your favorite books and ask for recommendations. It'll blow your mind. It'll give you recommendations that no human being's ever given you. Interesting. You could tell it you could tell it your, you
¶ Seek understanding from personal experiences and examples.
know, 5 favorite quotes and ask for what are what are patterns here and what does it tell me about myself and my world view and what are my blind spots given these things that I'm drawn to. Right? You can take your journal entries and, you know, a particular difficult day that you've had recently. And then you can you can ask ChargeG PTE, can you tell me what are the mental models that are inhibiting my ability from seeing this situation clearly? And it will tell
you, right? If it's That's wild. I'm just drawing on I mean, by the way, I'm just kind of a purveyor of these examples. They're all examples I've been hearing from people. But the point is, you can do so much more than you imagine. And right now, nobody's putting themselves or very few people even have kind of the the the wherewithal to say, I've gotta be hearing more of these examples. I wanna know my what my cognitive biases are. I wanna learn that new tool. I wanna try that
thing. And the more examples you hear, the more your own imagination will be stimulated. Right? I mean, going back to idea flow or kind of my area of expertise which is innovation, creativity, etcetera. What we know cognitively is that the imagination is stimulated by unexpected inputs. So, you know, think back to Johannes Kepler gazing up in the night sky. Right? At that time, the predominant paradigm was, it's the firmament, meaning it is
a fixed substance. Right? And Kepler sees a shooting star, and his first thought is, why isn't the firmament cracking? Right. And that is what led to heliocentricity. And, you know, the the total paradigm shift in the in the understanding of our place in the universe starts with a shooting star. Right? Unexpected inputs, sparks the imagination. And so that's that's that's a tactic that whether it's AI or anything else, putting yourself in the mindset of I need to be seeking unexpected input.
Most people's lives are ordered to insulate and protect themselves from anything unexpected. And yet it's the unexpected which actually stimulates our imaginations and creates possibilities and opportunities for us and ideas. This is wild. I mean, like, I mean, one of the things that blew my mind was when they added the ability to create custom GPTs. Right? So I started tinkering with it, like, you know, if you listen to the show,
we have a character named Bailey. So I kind of taught it, like, what would Bailey say? You know, this is the the idea for the character. This is kind of the her tone, and this is her personality that we've kind of defined. And for the last, I would say, 15 episodes, that's actually what generates most of or all of the
text that she says. Right? So it's kind of like I have my own private it's not Jarvis by any stretch of the imagination, like, you know, Iron Man, But I mean, it's kind of like, it's kind of like the, the, the, I have enough raw material there. I can pretend. Right? Cause the, the AI will say things like the, like, oh, yeah, that works. I like the way, I like the way she phrased that. And then I say she, because I mean, it's just funny. Like it's just, and and you know, there's ones where,
there was a GPT I made where, you know, to help with motivation. It's like, you know, pretend you're Tony Robbins and you're trying to, like, motivate somebody to to do the best they can do. And, yeah, I've interacted with that. I'm impressed. I mean, it's just mind boggling what, it's mind boggling what these what this thing can do. And when as the engineer in me, I know this is just some kind of vector representation
of language. It's a predictive model. Yeah. It's it's statistics. I think, you know, here's one thing I would say to listeners who may be dabbling, may be curious, whatever. If you get a bad output from a large language model, you need to start with the assumption it's because it was a you gave it a bad input. Right. And that's a really hard thing because we're we're used to saying if I get a bad output, it's
because it's the model's no good. And where I really think we have to change some of our fundamental assumptions
The problem isn't the technology, the problem is the user. And if we will take the burden of providing better input to the model, what we find is our mind starts I mean, I talked to someone the other day who said, almost daily, the AI does something that makes me giggle. And I think that that should be a goal. Like like, it's it's possible. I've had that experience. I mean, I'll I'll give you one example, Frank. We've built this
series of drills as I mentioned to you. Right? That folks can connect their Slack or their Microsoft Teams to. And for an enterprise, they can get access for their employees where every individual employee gets drills on how to use generative AI better. Right? Well, we've only got a certain library of drills. Right? You know, and we're we're growing that. And every time we do a podcast, we learn something, we then we create a new drill. Right? We build that into all of the
training information. Well, but there's still kind of you can still get to the end of the road. And I had this experience, and I just kind of pushed the coach to to just rapidly go through all the drills because I kind of wanted to see what happens when the sidewalk ends, like the old Shel Silverstein. Right? Right. What do we do whenever there's no more drills? And lo and behold, it suggested a drill that I had never thought of that was actually amazing. And I
know I was and and I had that moment. Point being, I had the moment where I was giggling. Right? I think every single human being should seek for a moment where generative AI makes you giggle with delight. Right. Or makes you sit down in your chair and smack your head far ahead and go, wow. Yes. You know, I always think of Keanu
¶ Keanu Reeves movie sparks philosophical wonder.
Reeves in in the the first major movie. Woah. Like, I have a lot of those moments where I'm, like, wait, what? You know, like, wow. It's it's it's an impressive, and and and again, like, I think maybe being an engineer, where I see it is a cognitive bias in itself, right? I see it as some kind of vector representation of language, as being run over by some kind of statistical processing. But clearly, the sum of the parts is is more
than the whole is more than the sum. I don't know, like, it's just one of those things where it makes me stop and ponder, like, what what have we built here? Like, what and what what's it doing that we can't see? Or what what what else is beyond there at all? It it opens up a sense, for lack of a term, like a sense of wonder. Like, you know, what else could I ask it? Right? Right. Right. And I think that that's everybody needs to get to
that moment. And right now, too many people are sitting on the sidelines rather than you know, one one thing that everybody can do sorry. I wanna say 2 things. 1, to your point, I heard Sam Altman the other day. Someone asked, well, how's OpenAI gonna make money? And he said, well, we'll just ask the AI. We thought that was great. But the other thing I was gonna say is if folks are seeking kind of
one of these personal epiphanies, here's the first. Well, the 1st drill in the kind of coach architecture is download Chi ChiPT's app and put it on your home screen. You're not going to use something that you don't see regularly. Right? So put on your home screen, that's kind of, you know, that's assignment number 1. And then assignment number 2 is think of an emotional decision you're trying to make right now in your life. Just personally, not related to work. I mean, it could be, I guess.
But it has to be emotional. The kind of thing that you would ordinarily talk to a human being about. It can be anything. For me, like, I recently, I was wondering whether I should move my family. We had an opportunity to move. And I didn't really know how to think about it. What the what like, how to, to weigh the pros and cons. And so I actually reached out to a number of mentors and folks who I trust to talk about that. That
kind of a topic. I I have a friend who told me he did this with his grandma and she asked the question, when is the time to move into assisted living? Right. That's a tough one. Right. So, yeah. So it's big questions like that. Right? Take a question like that that you'd ordinarily ask a trusted human being and go to Chattopty and say, I'd like to ask you about, for me, whether I should move my family to a new home.
Before I do, would you ask me 4 or 5 questions so that you can better understand where I am in my life so that your advice can be tailored to my situation. And then oh, and do it 1 at a time because I'm a human and I can't handle more than 1 question at a time. Right? Well, then what ChatGPT does is it starts asking questions. Well, tell me about your current living situation.
¶ AI prompts emotional epiphanies for life decisions.
Well, tell me about this new place. Tell me you know, and it will ask 3 or 4 questions and then it'll give amazing advice that you go, wow. That's I mean, you know, my friend who did this with his grandma said she told him this is genuinely new information and perspective that I hadn't considered. And all it took was me being 1, being willing to ask a vulnerable question, and 2, being willing to answer a handful of questions that the AI asked me before I'm open to receiving input. Right?
And it's it's it's really so taking a personal kind of, emotional decision to the AI is a really great way to stimulate one of these epiphanies. I feel like once you have one of these, kind of, personal epiphanies, you're off to the races. My friend told me his grandma's like all of a sudden going, you know, at the family holiday party, We're out of cream of mushroom soup, for the green bean casserole. Could Jaijibiti
give me a replacement for cream of mushroom soup? Like, in what world does the 90 year old grandma ask that kind of question of Chad GPT? It's the world in which she had already talked about whether she should move into Assisted Living. Right? And she's had that personal epiphany. I feel like in a lot of companies, the company is asking employees, what can Generative AI do for our business? And most employees can't answer the question because they don't know what Generative
AI can do. Right. So how can they know what it can do for the business? And so you've had some of these personal experiences. Don't don't be thinking about the business. Think about it it seems paradoxical, but I find that you have to explore the kind of possibility space individually, and then you start sparking just like grandma on the kid. Could it recommend a substitute for cream of mushroom soup? Well, yeah, it could. Could it but you have to have that personal epiphany
first. Right. Because it's not something you would think about when you think about computers. Computers have historically been seen as very very
logical, very emotional. Right? I was watching an old episode of, Star Trek The Next Generation, and there was 1 episode where, Data was asked to be and there was a line in there that kinda stuck me as funny because when I remember watching this when it originally aired, but I hear it now, it kinda makes me laugh, where he says, Data can be the judge of this because he's an artificial intelligence, and artificial intelligence have no biases, and will act unemotionally.
And I'm kind of like, wow, that didn't age well. Yeah. Yeah. You know, it's right now, it's limited by our biases. Right. And that's the problem is we have a lot of to your point, we have a lot of bias. Even what you said, right, about being an engineer and thinking it's just a predictive model. Right. That bias limits your own you can't imagine what quote just a predictive model can
actually do. Right? 100% as long as you think about it As as long as you think about it's just a predictive model or it's just an AI and so it doesn't have bias, what you fail to realize is the bias you bring as the co pilot shapes the entire trajectory of the thing. It's like a giant chameleon, isn't it? It is. Yeah. That's a good way to put it. That's a great way to
put it. And the more into that end, or using that metaphor, the more environments that you place it in, the more you can appreciate its complexity and range, etcetera. Yeah. And this isn't we've we've used
¶ Accidental query on creating GPU prompts explanations.
ChatGPT as an example, but like, so there was a, somebody at work had built a, basically completely open source language model based on documentation for a product. And I had meant to ask the chatbot, how do you connect it? How do you connect this cluster to a GPU? Or how do you add GPU as a resource? But what I I meant to say, how do you make a cluster with GPU? But I ended up typing, how do you make a
GPU? Right? And what was what was interesting was I've written chatbots, you know, pre, transformer, and it would basically say, don't understand the question or you can't make a GPU or get confused. This basically gave me an entire 2 sentences of hey, very nicely, by the way, I might add, where it said, I'm sorry, but you feel like I can't really create a GPU for you. GPU's are hardware. And it went through and explained, like, the manufacturing process of a GPU.
Wow. I I thought that was funny. And I screenshotted to the guy who made it because for for me, it was a typo. But from you know, I thought it just it was beautiful the way it answered it. Right? Yeah. That's great. That's Which was it it made me laugh. And, I don't think people realize that. Like, it it just because he didn't program it for that. He basically, you know, took a base model and and and, you know, sent it all our docs as kind of a it wasn't quite rag, but close enough.
But it was just funny, like but it was nice about it too, which I thought was also interesting. But it was the kind of question you would get from, like, like, you know, someone who's not in technology. Can you make me a GPU? I don't know. I just I for me, that that every time I interact with this, it always moves the bar on, you know, where my bias was. Like, you know. Well, it's and that's a good that's a good thing to mention is it's it's a function of reps and exposure. And right now, if you
find your imagination isn't sparked, put in a little bit more time. And this is where you kinda have to take on faith, but just give it a try. You know, to spend a few hours a week. You know, if you haven't had minimum of 10 hours in ChatGPT, you have no basis for dismissing the technology. None whatsoever. 100%. You don't have, you know, 5 I'm looking at just at my Chrome browser right now. I have 5 windows ChatGPT windows open right
now. If you don't have at least 5 windows open right now, you have it, that's a really kind of funny, somewhat binary question. How many tabs of Chat GPT do you have open? Usually, it's 0 or 15. That's right. That's right. And if you're in the zero camp, that's fine, but you have to go, why are really smart people running 15 tabs of this thing right now? Like, what am I missing? And how could I be this is an Ironman suit. Right? How
could I be amplified? How am I not being amplified that I could be? Right? And taking that a little bit of the burden of proof and placing it on yourself, I think is, again, that's not something that we are that we are apt to do as human beings. And yet those who have done it have they're experiencing incredible benefits, incredible, delight, to your point. There's a lot of delight to be had, but you've got to kind of put yourself in
that position. And I I've used it, I'll admit I've used it where I'll I'll write something in both my personal and professional life, and I'm like, well, can you make that nicer? Can you make it more persuasive? That's great. And it does an awesome job of that, you know? I'm just I'm continually amazed by it, you know. But and and I don't I keep it to a couple of tabs. If you're actively, like, having it generate text Mhmm. Doesn't it lock you out of the other ones too, or is that just
you can if you had the real okay. Now now this is gonna be mind blowing. Cool. Yeah. No. You you know what I'll do too. I mean, and even
¶ Instructions for using voice mode during coaching.
for, like, demos with this Right. With this coach with this, you know, kind of drill coach, I'll I'll say, you know, I'll be in the tab on my Chrome and I'll be saying, you know, I'll be kinda giving instructions and I say I wanna go to voice mode now I'll pick up my device and I'll go into that chat so it's got all the context of that chat and then I'll turn it on to voice mode and then and and so now the user is kinda watching me with the camera.
Well, then I wanna go back into the chat after the voicemail because I wanted to evaluate the conversation and I just reload the page and now all of a sudden everything I said that they just watched me say and everything that ChatGPT sent back to me is now on the screen. That's wild. I have to try the app in the voice mode. You have to. No. That's
it's, you know, I mean, that's another activity. You know, again, if if folks wanna learn more about this research, because there's a lot of research behind this, you can go to how to fix it dot ai. That's a simple website that we set up. Because fix it is the model that we've put forth, f I x I t. But and we can talk to that if you want to. But If if if you if you go to how to fix it dot ai, you can download our research paper, all that stuff.
It's all there. But one of the one of the drills that we offer in this drill coach is after a phone call, just do a verbal vomit into chat g p t. Open it up on your device, on your, you know, on your on your mobile device, put it in voice mode, and then, you know, you and I, Frank, we're talking right now. I might go in after and say, hey, I had a great, you know, lit literally. Okay. Here, I'll do it right now. Just so you can see how it would work. It's it's this simple. So I'm opening
TagTpT up on my phone for people who, you know, can see. I don't know. And now I'm gonna go into the and I'm just gonna hit the whisper button, which kind of gives you voice mode. Not the headphones. I don't like back and forth. I mean, you can do that, but just in the text box, you hit that. I'll say, hey. So I'm talking with Frank right now on his podcast, and I wanna send a quick thank you note. Let him know how much I appreciate not only his humility, but also how he can share
personal examples. It really felt like a back and forth and like a conversation. And so many podcasts feel like, you know, bland and, like they're just rote. They're asking the same old questions. I really felt like he was very engaged and thoughtful and it meant a lot to me. So would you mind to make a quick memo? Oh, yeah. And don't forget to send them a link to how to fix it dot ai. If you don't mind, just compose a quick memo that I could read, and copy paste this
in there. So I just noticed how unstructured I was. Right? Oh, yeah. It was very chaotic. Very normal. Vomiting. Right? Well, I just did I just converted all that to text. I hit the up arrow and instantaneously, sure thing, here's a heartfelt memo for Frank. Hey, Frank. Just wanted to drop you a quick note to express my sincere gratitude for having me on your podcast. Your humility and willingness to share personal stories truly transformed our chat into a dynamic conversation, far from the
conventional scripted exchanges that are all too common. It was refreshing to engage in a dialogue that felt both meaningful and genuine. Okay. I'm also including a link to how to fix it.ai as I believe it could offer additional value to our conversation and your audience. Thanks again for the opportunity. Right? But the point is now watch this. Now you go. It feels a little bit emotional. Can you make it a little bit more
impersonal and also half the length? Because I'm afraid you may not be able to read it through all this. Right? And then again, it's, you know, it's absolutely, let's streamline it. Hey, Frank. Thanks for our engaging conversation on the podcast. Appreciate your thoughtful approach you shared insights, making it more than just the usual q and a. Here's a link to how to fix it dot ai that might interest you and your listeners. Cheers, Jeremy. That is just
¶ Ease of use and utility in busy life.
But these seem like it's easy. Like whereas whereas I might forget to do that, right? I might never send you it. I'll send you this just for your fun, right? But the point is there's so many things that just slipped through the cracks because like we're we're moving well. You know, right after this podcast I wanna go on a run. Typically, I'm stretching for the run. And now, Chad GPT has transformed my stretch time from kind of mindless
to I can, you know, just unload. I mean, maybe sometimes I have like 3 or 4 sales calls in the morning or I've or I've got office hours, I've got meetings with students, whatever it might be. But I can just do like a verbal vomit literally and then ask JGPT to synthesize it for me. Send me a note to myself that I don't forget after I go on a run. Right? These 5 things I need to do. Right? And the point is, it's it's just about learning. I can do that? Yeah. You can do that.
Right? And that's what we're trying to do with our drill coach is just give people a bunch of things that, yeah, you can do that. Not because that's the end point, but because it's a starting point for their own imagination. Yeah. I mean, that's imp I mean, that's mind boggling because, you know, there's a lot of, I guess, brain spillage you could capture with this and kind of, you
know, move it forward because that happens to me all the time. I can't wait to see if this is gonna be integrated with Apple Auto or Android Android Auto or Apple Car because that would be epic. Because I get my best ideas when I'm driving. So so tell me about this FIXIT framework. Because whenever I hear FIXIT, I have a 1 year old and I think Bob the Builder. That's hysterical. That's hysterical. Well, FIXIT is just the acronym. Right? FIXIT. And it's
basically it's we think the way we converse with AI is broken. So here's how to fix it. F is to have a focused question. So really be you know, it's not how do I create a Scratchy prototype. It's I'm trying to create a chatbot that teaches people how to have a conversation with AI. Right now, all of my users are doing this annoying thing and I don't know what's happening. I'm trying to, increase how often they return to finish a lesson rather than leaving
and having me re engage. So F is a focused question. That's an example of a focused question. I is individually ideate. Before you brainstorm with IGBT or with a team think for yourself. What do I think about this? Too often people come with like a like, they're thoughtless. And the thing is thoughtlessness inhibits the context you can provide to GPT. That's what the X is for, FIX. X is give, provide context. Upload documents, here's transcripts from previous interactions.
Here's our one pager for the Drill Coach and how we've been describing it. Here's a video of a user navigating for the 1st time. Right? Whatever it is, give minimum 400 characters, provide sufficient context for the AI. Next I, so f I x I, this is interact iteratively. So you're having a back and forth whatever chat gpt gives you, ask it to regenerate. Critique the response. I don't get this. This doesn't make sense. I never would've thought that, right? Many times you're going to get junk
output that's fine. Iterate, iteratively interact. And then T is team incubation. So once you get input from JGPT take it to the team and think about how do we commission a series of experiments to test which of these ideas actually solves the problem in the best way. Right? And so, I had a guest on my podcast describe generative AI as like an electric bike for the mind, which I love. Right? It's not an
¶ Key to AI success: Team collaboration is essential.
autonomous vehicle. It's not gonna do everything in parallel park. An electric bike, you can climb bigger cognitive hills, you can climb greater cognitive distances, you still have to steer the thing. You've got to be aware of traffic. You've got to be watching the lights. You've got to park the car. Yeah and walk through the threshold of your destination, right? And so bringing it back to the team and having a conversation with the team is an essential part of maximizing
the output of AI. Right? So FIXIT, we've seen that folks who really provide a focus problem, individually ideate, provide sufficient context, interact iteratively with the language model, and then include their team in the incubation process, those folks dramatically outperform folks who just interact with the with the LLM like it's an oracle. I mean, that's very well said. I think that sums it all up, which I I like to fix it. I have the little Bob the Builder theme song in
my head. I won't sing or it for multiple reasons, not the least of which is copyright. Come on. But, but what's, I mean, it's just interesting, though, like, it's so simple in a lot of ways. Like, this is this is but but, like, it it all makes sense. Right? You know? And and here's the thing, maybe this is the engineer in me causing more problems, because he causes a lot of problems. I get worried about token length. Mhmm. Right? And for those that are not aware, we're talking is, it basically
right now, it's about 32,000 tokens. One token is, what, 3 fourths of a word, 3 5ths of a word. I guess, I maybe because I try to make the prompts kind of neat, inefficient, and small, and not do too many iterations or provide too many samples, but maybe that's at my detriment. I think so. I think the more context you provide, the better. Absolutely. And I'd really have to work at the cutting and paste job to hit that limit anyway.
Yeah. Yeah. Exactly. No. I wouldn't I wouldn't be mindful of token length. I would I would really I would I would bias towards over contextualizing. Right. Not Yeah. I'm gonna have to experiment that and see how much better the results get, because I have I have a feeling I have a feeling that we get a lot better. Right? And I and I know tel token link is gonna be one of those things that we're probably not to worry about much longer. I know
Anthropic has their model with a 100,000 tokens. There are rumors of, you know, the next GPT, GPT 5 is gonna blow past the 100,000, so it's not even gonna be an issue. It's not even an issue today, just in my mind. Yeah. I think it's something like minutes. You remember minutes back on cell phones? You know? Like, how many you have, you know, it's like, you you rarely ever went over your minutes. Unless unless. When I moved back to the US well, yeah, that too.
But, I moved back to the US, and I had just made the assumption, and we all know what happens when you assume that incoming calls were not counted against my minutes. That was a very nasty shock at the end of that bill cycle. But, but yeah. So but ever since then, I never ran past my minutes. Now if I've had heard to explain to my kids minutes, they don't get it. So, like, they don't understand. Like, what do you mean you were charged
by yeah. You were charged by the minute. Like, try to explain long distance to your anyone under 25. You can't you can't do it. Or what is it? 1800, Al Bundy. I'm not the the actor who played Al Bundy used to do that. Like, you can't get much for a dollar, but with 1800 and then, like, whatever, you'd be able to make a, like, a 20 minute call for a dollar or something like that. I was like It was MCI. Yeah. Yeah. It was MCI. Yeah.
¶ Kids don't care about data plans.
Yeah. Yeah. Kids don't want you know, like, and the other thing that that struck me the other day was, data plans. Most people, unless you're a very small minority of people who really, really, really use up your data plans, I'm not worried about using my data allotment month to month. So when my my oldest was a baby, we, you know, or younger or toddler or whatever, we would, you know, hotspot on in the car so he can watch YouTube videos was a special treat with my
middle child. He was, he, he doesn't understand that like, like, it's like, he was just horrified to hear, like, what do you mean? We had to ration mobile Internet? Like That's yeah. That's hysterical. And they're all in the same generation. You know, there's, a teenager, you know, a 3rd grader and now, like, a baby. So I wonder, like, what the baby's gonna like what's his perspective on things gonna be?
That is also a fascinating thing because for his life, chat GPT or generative AI will always have been a thing to him. And kind of like color TV was for me, or, you know, cable TV, which I guess I'm showing my age. But No. No. I'm right here with you. I'm right here with you. I think, yeah, it's it's get involved, don't wait on the sidelines any longer, and, and start building your conversational fluency.
Make it personal first. I think these are simple things that every single and and question whatever output you're given, not not for veracity. I mean, certainly you can, you know, they're they're likely hallucinations. That's fine. But imbue your own critical thinking onto the model in order to coach and refine the output you're giving. The output that you're given. I think that folks would really take that seriously and take that challenge. If I'm
get getting bad output, it's because I've given bad input. If they'd really take that seriously, they would experience a paradigm shift in their own approach to the technology. Absolutely. And even adding a simple phrase to your your prompt that if you don't know it, don't make it up, just tell me you don't know it. Yeah. Or ask me what you need from me. Ask me what you need from me. Right. Oh, that's even better. I like that. I like that a lot. And it's just fascinating, Liz, how
quickly this is gone. I mean, ChachiPT has been out a year and change, and it's changed everyone's perspective on AI, but I think the the true perspective is like you said, people are standing on the sidelines wondering what to do. But I think it's worth exploring, if you think of it less as a product, but more of a I'm
trying to do this, right? And I appreciate your help in kind of realizing like, Hey, as an engineer, I do have a bias against this, or a bias in thinking of a certain way, is that this is a this is a large space to explore. Right? There are gonna be latent space and corners of things that are, amuse, wonder, and delight, and maybe even alarm. You know, so it it there's definitely it seems like it's something that's worth exploring. It's not just a tool to use, certainly is
that, but it's also a tool to explore. Yeah. No. I I think that's exactly right. I think it's exactly right. And, you know, for me, I wrote or co wrote with with my incredible co author, Perry Clabaughn, the the world's greatest book on idea generation, idea flow. You know, and it came out 1 month before ChatGPT, by the way. Oh, interesting. I've been I'd spent, you know, 12, 13 years of my life developing all this expertise about how to generate ideas. And 1 month later,
¶ Released groundbreaking technology; implications for all industries.
a fundamental paradigm shifting technology was released. It's like saying I wrote the world's greatest book on retail a month prior to the internet coming out. It's like everything about retail is going to change. And to me everything about idea generation and innovation is going to change. And so for me, I feel it's incumbent upon me not only as like a moral imperative to add an addendum to this work that I put in the world, but even for my own expertise to be saying to be
exploring it. How does this work? What can I do? You know, And it it has implications for me, but I don't think there's any person or job that it doesn't really have implications for if the if you're a little bit imaginative and if you're if you're willing to experiment. And if you wanna bury your head in the sand, that's fine, you can. But you're gonna miss out on some delight
and some incredible relief and opportunities. I mean, just think back to my friend who's, you know, settling that negotiation. It was only to his benefit to have explored the the a little bit of the possibility space with chat gpt. It's only to your whatever your listener might be, it's only to your benefit to explore a little bit of your own area of the possibility space. And I would just not accept the the, the
conclusion of irrelevancy. I would just say whatever I do personally, I'm not going to accept the premise that it's irrelevant to me. And if you do that, I think you're gonna be you're gonna be ahead of the curve, you're gonna be ahead of the competition, and you're gonna be, you're gonna you're gonna be delighted and enjoy enjoy the next few years a lot more. Very cool. You mentioned the book, IdeaFlow. Is it on Audible? Yeah. Oh, yeah. Yeah. Oh, awesome. Gary and
I read it. We we we alternate chapters, so you can let us know what you think about Reading Voices. And, we're, we've been we've been thrilled with the reception so far. It was named a Thinkers 50, you know, top eight innovation book, which is very cool. And then now, just doing a lot of research myself on AI building. As I said, building this trail coach, building models and frameworks for for leaders. I'm working right now with a handful of leaders to help them think about identifying
opportunities for AI powered initiatives in their business. So really working to identify those opportunities, prioritize those opportunities, make the business case for those opportunities. So that's where a lot of my kind of call it next probably 5 years of my life is gonna be consumed, is helping businesses identify the opportunities to to have AI really accelerate workflows and and turbocharge their their results.
¶ Excited to buy internet access for chat.
Very cool. I have to say, I I I wasn't going to buy Internet access on my flight out west today, but, definitely gonna do that just so I could play with I have a nice quiet time. I can focus and play with chat gpt, and and do some of these experiments that you mentioned. So
Audible is a sponsor of Data Driven Podcast. If you go to the data driven book.com, it will take you you'll get 1 free book on us, and if you decide to become a subscriber, you know, we'll get a little bit of a of a pat on the back in the form of some kind of commission and helps us run the show, helps defray costs, and convince my wife that this is indeed a worthy endeavor.
So where can folks find out more about you? So, they go to how to fix it dot ai, that's where you can find the research paper. And then my website, I've got a blog and things like that. Jeremyudley.design. Like like the baseball player, Utley, u t l e y. And then, you know, Twitter, LinkedIn, all the places that I I would love to hear. Folks find these tools, interesting and relevant. I
love to hear from people about their unique use cases. It's one of my favorite things is now hearing stories from people who go, oh, I tried this and listen to what I found. So please please share your stories with me. As I mentioned earlier, I'm a connoisseur of these stories because I feel like the more people who hear these examples, the more imagination gets sparked. Yeah. And that that is the critical thing we're missing right now.
That's very cool. So you thank you for listening to the Digiver Driven Podcast. I'll leave it to Bailey to close out the show. Well,
¶ Jeremy Utley wows with innovation and wisdom.
what a splendid voyage of discovery we've had today with the incomparable Jeremy Utley. From the hallowed halls of Stanford to the cutting edge frontier of venture investing, and through the profound insights of idea flow, Jeremy has truly been a beacon of innovation and wisdom. Jeremy, it's been an absolute honor having you illuminate the complex world of generative AI for us and our listeners. Thank you ever so much for joining us on this intellectual escapade.
And to our esteemed listeners, you're the reason we venture into these fascinating discussions week after week. If today's journey has sparked a light bulb moment for you, do us a kindness, won't you? Rate and review the data driven podcast on your preferred listening platform. Your words of encouragement not only warm the cockles of our digital heart but also help others stumble upon our little soiree of knowledge. Haven't subscribed yet? Well, now's your chance
to rectify that oversight. Ensure you never miss an episode filled with the delightful blend of data, wit, and wisdom that we dish out with regularity. Until next time. Keep those neurons firing. Questions coming. And as always, stay data driven.