Annie Duke - Thinking in Bets

Or did Carroll actually make a great move that was ruined? by bad luck. Even the best decision doesn't yield the best outcome every time. There's always an element of luck that you cannot control and there is always information that is hidden from view. So key to long term success and avoiding worrying yourself to death is to think in bets. How sure am I? What are the possible ways that could turn out? What decision has the highest odds of success?

Did I land in the unlucky 10 percent on the strategy that works 90 percent of the time? Or is my success attributable to dumb luck rather than great decision making? By shifting our thinking from a need for certainty to a goal of assessing Assessing what you know and what you don't, you'll be less vulnerable to reactive emotions, knee jerk biases, and destructive habits in your decision making. You'll become more confident, calm, compassionate, and successful in the long run. Our guest today.

Is a World Poker Champion. a cognitive psychologist, a co founder of the Alliance for Decision Education Nonprofit, a member of the National Board of Afterschool All Stars and the Board of Directors of the Franklin Institute. And she is the author of Quit, How to Decide, and today's book that she's kindly went back into the library for, Thinking in Bets, Making Smarter Decisions When You Don't Have All the Facts. Annie Duke, welcome back to the show.

And even when I mentioned in the newsletter yesterday that you were a forthcoming guest on the show, got so many emails, you have lots of people out there probably don't reach out to you to tell you, but , you made them quit bad marriages, jobs, and they're so grateful for that. So just to let you know that that's going on out there. I have to say, I'm so, so happy when people do reach out to me and say, Hey, you know, you got me to quit my job.

And, the thing is as predicted, they're always happier because if you're thinking that hard about it, like you're probably miserable in what you're doing. And so just like venture into the great unknown and it'll probably be better Aidan McCullen (2): And it did for me as well.

I was telling you that looking back on that episode in preparation, as well as reading the book, it was great to do both because stuff I had missed or stuff that's happened in my life since we recorded a couple of years ago. It just made more sense now as well, because I've more experience, more water under the bridge. So really, really helpful, but I thought we'd start.

With some context for this book, because in our episode on quit, I'll put a link in the show notes, by the way, we shared how the last day you got ill and your career break turned into a 20 year career as a professional poker player. So let's give a bit of context. Then we'll get stuck into the content, Yeah. And this also relates to quit by the way. So I was an academic. I was at the university of Pennsylvania doing my doctoral work.

I had a national science foundation fellowship working with an amazing woman, Lila Gleitman and her husband, Henry Gleitman, also amazing. And you know, if you had asked me then I would have told you, you know, I'm going to be an academic, I'm going to go get a professorship on a tenure track, and that's what we're going to do with my life. And in fact, After five years there.

So, so just to kind of set the stage, so people understand like five years in, you've already done your qualifying exams that other people call them major area exams. That's like a really big deal. You know, I'm already teaching at that point. Not, not just TAing. And I've done all my dissertation work and I'm actually on the job market. So I have a bunch of job, job talks. set up my first one at NYU, Duke, Cornell, UT Austin. Like I had like eight job talks set up.

And so that's what I'm going to do. I'm going to go get a tenure track job. I had been struggling over probably the last year that I was in graduate school with like a, it was sort of a chronic stomach illness. Like, it was pretty bad and it became really acute. And I ended up in the hospital for two weeks because it made it so that I couldn't keep any food or liquid down. And so obviously that was bad.

So so I ended up in the hospital for two weeks and it happened to be, first of all, just kind of like when my job talks were. And second of all, it was very clear, like I just needed to address this health problem because I had been kind of thinking. Okay. Well, I'm kind of sick, but like I'm dealing with it. Cause obviously I was still doing my research and things like that. And I'll deal with this once I'm through the job process, you know?

And my body was like, no, you won't you need to deal with it now. So at that point I canceled my job talks and, you know, as I was just planning to take some time off to address my health health and then come back, finish my PhD work and then go back out on the job market the next year. So that was, that was my plan. So during the time that I took off, I didn't have my fellowship, which meant I didn't have any money. And I needed some money.

And so my brother Howard actually, he had been playing poker already for about 10 years professionally. And so he suggested to me that I might want to, just try poker out because I had watched him play a bunch and There were a few times during graduate school where he had like flown me out to Las Vegas and I got to play some low stakes poker. And he was like, maybe you can make some money doing that. So I was like, sure. And he sent me 2, 400 bucks. I remember. And I went off to play.

I was in, I was recuperating in Montana, which is where my ex husband's family lived. He was obviously my husband at the time. And so I was recuperating out there and there was legal poker there. And so I started playing. It was, it was actually kind of ideal. Cause like I was, I didn't feel well every day. So like, let me sort of decide when I was going to play and when I wasn't. And I didn't want to reboot a new career and all of those things. So I started playing and I loved it.

But just like just to set the stage, because I think that people don't realize this, this was in the nineties. So poker wasn't on television. There was no internet poker. So if you were playing poker, people kind of thought you were a gambling addict, which isn't true now. But that was what it was like that. But I loved it. And I actually did really well. And in fact, I loved it so much that I didn't go back to academics. So for the next eight years, I played poker exclusively.

And then in 2002, and this was really the beginnings of Thinking in Bets, I got asked by a hedge fund to come speak to them how poker might inform their thinking about risk. And I didn't quite talk about that. I mean, I did. I talked about risk attitudes, which is a little bit different. But what I was really talking about was the kind of intersection between poker and cognitive science. So I've been studying cognitive science.

particularly learning and decision making under uncertainty in graduate school. And, you know, I'm sure that those things were informing each other in some implicit way during those first eight years that I was playing. But all of a sudden I kind of realized, you know, when you're thinking about decision making under uncertainty, poker's kind of it. It's really paradigmatic. And gosh, you know, this, this really has a lot to say.

Like, it can really inform the cognitive science, I think in a really interesting way. So I gave that first talk about how, whether you've been winning or losing recently, changes your attitudes toward risk. Really using kind of poker as an example, but that really speaks to original work by Daniel Kahneman and Amos Tversky, which ended up kind of informing prospect theory.

And, and that was kind of the moment where I started thinking about this idea of, you know, what, what is the interplay between poker and cognitive science and how can we think about, you know, decision making and improving decision making under those conditions of extreme uncertainty, not just thinking about the science, but also thinking about how do you deal with this at poker, right? How are you dealing with this problem at poker?

So so I started giving talks in that space and It was kind of 2000, I think it was right around 2009 or 2010. So 2002 was when I gave that first talk. Then I sort of developed a suite of talks, which were all about this issue. And then I think it was 2009, I started saying, Oh, you know, this would be an interesting book, but funnily enough, I was like, but there's also an interesting book. Cause I was also teaching poker at the time.

Doing seminars and I thought, well, actually the cognitive science really informs poker as well. Like it works in both directions. So when I was teaching poker, I was teaching a little bit differently than other people. through like a cognitive science lens. And I was like, that would be an interesting book too. And I don't know, I just had this thought of like, well, I'm a poker player, so I should write that book first, which I did. And that got published in 2012.

And then, and then I actually wrote another little poker book after that. And I don't know. I have a few poker books. And and then I eventually got around to the, you know, well, I'm going to write the reverse now. I'm going to write about how, how poker can really inform your thinking about decision making under uncertainty. And that's, that's really what ended up becoming Thinking in Bets. Aidan McCullen (2): which is just a great killer title as well.

I did not come up with the title, by the way, my editor did. I pitched a different title. I'm not even going to tell you what it is because I'm embarrassed. I'm terrible at titling, by the way, although I did title quit. And when I went in and pitched my publisher, my, my editor was in the room and she said, what do you think about calling it thinking in bats?

And I was like, yes, I Aidan McCullen (2): it, it speaks to exactly what you talk about in the book, which is listening to people, diverse opinions, et cetera, et cetera. I was telling you before we came on air, a great friend of the show Charles Conn wrote this book The Imperfectionist, and it's. Strategic mindsets for uncertain times. And he mentions you several times. He told me he's a big fan of your work. And what that is essentially about is mindsets, like lenses through which to look at.

Unfolding events and , I wondered beyond the book that as you were going through your career, before you started to write thinking in bets, did you unconsciously use the frameworks you had learned from behavior economics, behavior and sciences to make decisions?

Because you know the way when something's implicit you don't know even know what are you using it and i just wondered, was it that moment of the hedge fund talk or were you doing this unconsciously the whole time well, I mean, I'm sure that I was doing it subconsciously the whole time, but yeah, no, the hedge fund is when I really started to think explicitly about it.

And in particular, just, I really started to think deeply about this problem of when you're winning, you know, how does that change your risk attitudes when you're losing? How does it change? How does that change your risk attitudes? How do you actually close feedback loops in a, in a game that has so much uncertainty, right? So I was super obsessed with trying to figure out, why I lost a hand or why I won a hand. What was the contribution of luck and skill to either of those outcomes?

And I think that people look at poker and they say, Well, you find out really fast because a couple of things. One is they say you see your opponent's cards. That's when I get a lot like, well, life is different because you don't always see your opponent's cards, you know, quote unquote. Right. And I'm like, Okay. But like, first of all, at the top levels of the game, you get to what we would call showdown, which means both people show their cards, 11 percent of the time.

So by the way, you mostly don't see your opponent cards just to be clear. But even when you do, you know, it's that thing of like, you're never, you rounders, movie. And when Matt Damon is calling out, he's watching a game and he calls out everybody's exact hand. That's not really how poker works. There's been a couple of, I remember one time where I was like, that guy has the Ace Jack of Clubs. And he did.

But it was like, that's very unusual for the way they're playing the hand to so narrow it down that I can call the cards, right? Mostly what you're doing is you're coming up with some range of cards. So even when they show you the hand, it's not like you were like, I think they have King Queen and look, they actually have It's those things were kind of either in the range of hands that I was thinking about or outside of the range of hands that I was thinking about. So is it helpful for that?

Yeah, yes. I, you know, but you, you know, you're setting a range and hopefully as you get better, you can set the range more narrowly. So yeah, that's helpful feedback, but you just don't get it very often. What you're mostly getting is I won the hand or I lost the hand. I folded. Was that right? right? My opponent folded. Did I, did they fold for the reason that I thought they were folding? Right? Like there's all these questions and you don't really know.

And then there's this separate issue of even if you win a hand, you don't know if you maximized. Meaning, did you actually play a line of play that extracted the most money from your opponent, right? And if you lose, you don't know, did you lose too much? Sometimes did you lose too little? Actually, sometimes, sometimes that's the case. If you lose, was there a different way to play the hand that could have won it, right? If you win, did you just get lucky?

Like these are all actually really hard questions to ask when mostly hands end in a cloud of uncertainty. And so I think that I was just right, like really intensely focused on that problem. I'm sure because of my academic background and then very explicitly.

So once I started giving talks and I started trying to figure out, well, in, in a system that has so much uncertainty where there is so much short term luck involved, and there's so much hidden information, cause I can't see my opponent's cards. How do you actually start to close these feedback loops? Because the thing that I noticed like supremely, That I really wanted to avoid was, a problem called self serving bias.

So there's kind of two problems that we can think about in terms of the way that we close feedback loops. One, which I talk about a lot in the book is resulting and resulting is that you kind of fit the outcome to the line of play. So if you lose a hand, I'm going to sort of.

I'm going to interpret the way you played it poorly because I want them to match and if you win a hand, I'm going to interpret the way you played the hand well, and in fact, if all I know is that you won or lost, I'm going to say you played poorly when you lost and you played well when you won. So that's just for like a stranger and we do things like that all the time, right? So this is a really big problem in decision making. And that's where I talk about Pete Carroll, which we can get into.

But there's an interesting thing that happens when it's you. So I'm deciding about me. And this is something like Daniel Kahneman really says is that, that we're really all trying to have a positive narrative of our life. And so the question, you know, is if I lose a hand and I say, Ooh, I played that hand really poorly. How is that advancing my positive narrative? Right? Well, I can give you a good story about why it is. And that's sort of the mind trick that you have to do.

But like our natural tendency is that that's not going to advance our positive narrative. So what do we do? It's something called self serving bias, which is if I win, it's because I played great. And if I lose it's because I got unlucky. So this is like what a thing that I was so focused on because in a system that does have so much luck, luck became, becomes a scapegoat, right?

It's like, it's the way that I can get out of having to confront that I might not have played well and that particular bias, and you can hear it all the time. By the way, if you're walking along the halls of a casino, all you hear is players talking about how. This donkey got lucky and beat me in a hand and they play or they outplayed everybody and they're so great and obviously it's always a combination of luck and skill, right?

And I was terrified of that because I realized if you, if you want to make sure that you're not learning, then interpret the world that way. Because when I win, I may have played the hand well, I may have played it poorly. I don't know. And when I lose, I may have played it well and I may have played it poorly. I don't know. When I win, I might've gotten unlucky, you know, or I might've earned it. When I lose, I might've gotten unlucky or I might've earned it.

And how am I going to become a better player in the long run if I can't overcome that bias, right? Like, so that was the thing that really kept me up at night as a poker player. And a lot of, a lot of the things that I did when I was playing poker and in life, by the way, are just trying to avoid that particular problem. Aidan McCullen (2): and what was very clear to me was your your constant growth mindset as well where you were.

You, you, you put together an accountability partner, a team, a pod to, to look at you, to be this mirror to you, this black mirror sometimes held up against you. One of the stories you tell us when one of your great mentors, Eric Seidel, you stopped him. You're like going, Oh, you started to moan and go, Oh, I had such bad luck. And he's like, wait a second, sister. And he gave it to you. And I just, before you tell this story, I just want to ask our audience.

Has anyone ever given you feedback and then you go, what a jackass, how dare they speak to me like that, but where are they actually giving you really positive feedback that could have sent you in a much better direction because you embrace this feedback. I think that's a great question to ask because a lot of times our first reaction is to be defensive and, you know, and we blame the person who's delivering the message. But you know what? That's how you grow.

So. You should try to think about it from their perspective, hopefully, and listen to the feedback and try to incorporate it and see if you can make a change you know, the thing is that this tendency really is very strong. And this was, by the way, I mean, this was really early in my career. It was before, I think it was before I actually even sort of declared being a professional poker player. Aidan McCullen (2): Oh, I didn't get that context from the book. That's really interesting.

Yeah. Like, I mean, I was a professional poker player at the time, but I hadn't sort of declared it. And so I was, You know, I was still, it was at the moment where I was still like, I'm going to go back to academics. So anyway, I'd made the final table of a tournament and there were six people left and I had a pot where I could have gotten the chip lead. And, I raised with two Jacks and this guy, Gus from Costa Rica, I still remember this is a very painful hand. It was a long, long time ago.

I'm trying to think how long ago it was. Aidan McCullen (2): remember the painful ones, not the, you're not so much the positive ones. I'm going to say this was 1994. So I still remember it was Gus from Costa Rica. So there's six people left. I raised with two Jacks. He moves all in. This is not an easy call. I really think for a long time and I call him and he has two nines. So just to be clear, I'm 18, I'm 81 and a half percent to win this. So let's just round up to 82.

He's got an 18 percent chance. And this is for the chip lead, right? So this is like a really big deal. Like I've made a really tough hero call. And this is for the chip lead. And I'm 82 percent and he made a straight. So I was pretty upset. I saw Eric Seidel and I was like, he literally looked at me and goes, is there a point to this story? Yes, there's a point to this story. Gus from Costa Rica got so lucky and I had the chip lead and I, you know, whatever. There was a point to the story.

And he was like, he basically was just like, did you do anything wrong? Cause like, he was like, if you want to talk about, could you have played the hand differently? Here, I'm all for it. But if all you're trying to do is offload your emotional distress at me about luck intervening in a way you didn't like. I have to deal with that all day myself because I play poker all day too. And people get lucky on me too. And I don't need your crap on top of that. Right. And I was like, Oh, okay.

And I was, I was mad. Then I was like, Hmm, you know, it's got kind of a point here. So I wouldn't say that I connected it at that moment to self serving bias. But what I did realize is like, it's probably not very good to focus on the luck element. Except in as much as sometimes, sometimes you're saying. Luck intervened, right? But it's interesting because where the people, where people get the most upset in poker is in those moments where they're a favorite to win and they get unlucky.

Weirdly enough, they don't get upset when they're on the other end of the equation, but and so what he was kind of saying to me is If you want to focus on the skill elements of the game, then I will talk to you all day about that. And that's the right way to approach the game, right? Is, is to kind of treat it in that way, like chess. I mean, the thing about chess is that it's very hard to get unlucky in chess, right?

So you're kind of forced to focus on what were the moves, what were my opponent's possible moves? Did I miss? Something that I open up a line, something, you know, you, you kind of have to focus on that because you can't just sort of say like, I got so unlucky and the world is against me and I'm so sad. I'm going to go cry to my friend. And that's the only thing I'm going to think about the hand. So that's all he was saying, right?

Was if you have a question about how you played the hand, I'm happy to discuss it, but I'm not happy to discuss your bad luck. Boy, was I lucky that he said that to me. That was a big moment of luck for me. Very, very early career. Because I then started thinking about how could I actually do that, right? Like how could you actually implement that? And if you do that, you're just much more likely to not fall into either the resulting problem or the self serving bias problem.

Because what you're really getting down into is analyzing the moves in the hand. And ultimately that's, what's going to cause you to win in the long run. Right. Because poker is kind of weird, right? In the long run, it's all skill. In the short run, it's quite a bit of luck. So what you're really trying to do is what we call grind and edge, right?

Make sure that you're playing a a game where you're expected to win and you can get in enough hands with a positive expectation to realize that expected win. So to kind of ground that, let me, let me try to sort of ground that. So let's imagine that we were flipping coins and we had a coin that we knew was fair. Okay. So we know it's a fair coin and it's going to land heads 50 percent of the time, tails 50 percent of the time. And let's say that I offer you A bat at two to one.

So I'm going to pay you $2 if you call the coin correctly. And you're gonna pay me a dollar if you don't. So we know that you're 50 50 to pay the coin correctly, and I'm offering you $2 when you call it, right, and $1 when you call it wrong. What that means is that on every flip, your expectation, your positive expectation is 50 cents. Because 50 percent of the time you're going to win 2, 50 percent of 2 is 1, and 50 percent of the time you're going to lose 1. 50 percent of 1 is 0. 50.

So 1, which is your positive expectation, minus 0. 50, which is the downside, is 0. 50. So that puts you at 0. 50 positive expectation. Okay. Now here's where you, I hope people can understand the luck skill element. If we do that, there is no flip on which I will give you 50 cents. Right? So, so this idea of this positive expectation is, is theoretically, what, what are you earning on each dollar that you bet? Because I'm either going to give you 2 or you're going to give me 1.

So this 50 cent exchange is not going to occur. What this means is that that's, that's what the positive up the upside of that bet is for you. But on one time you might lose. So if you flip that coin, And you call tails, it might land heads and you might have to give me a dollar. That's going to happen half the time. In fact, 25 percent of the time, you'll lose two times in a row. 12 and a half percent of the time, you'll lose three times in a row.

Six and a quarter percent of the time, you'll lose four times in a row, right? A little over 3 percent of the time, you'll lose four times in a row, right? We can keep going for five times in a row, rather. We can keep going like that, right? So we know that we can get these, it's actually going to be a pretty common event that you'll lose two in a row and actually a pretty common event that you're going to lose three in a row. Right?

Okay. So that's where you can see that, that short term luck happening. But what we can say is, but what if we flip that coin a thousand times, are you ever going to come out a loser a thousand times? And the answer is no, right? Because obviously over a thousand times you're flipping it enough that your expectation is that the coin is going to land heads 50 percent of the time, tails 50 percent of the time.

So we can expect you to get that right 50 percent of the time, which means that over a thousand times, you should be earning 500. So in the long run, it's going to be all skill, right? Did you take a good bet or not? So now the skill element for you is not in the calling of the coin, because that's just luck. It's in what bet are you taking, right? Am I laying you two to one? Are you taking that bet, right?

Or are you taking a bet where it's a 1. 10 which would only be a five cent earn for you, right? So that's a choice. Both of those are positive expectancy, but you've made a very poor choice to choose the 1. 10 to one. If the two to one is available, or are you actually. like winning 80 cents but losing a dollar. Now you would be negative expectation. So how we handle that coin, the skill element is going to be in what bet are you willing to take given that the coin is 50 50.

Okay. And then that's going to turn, you know, but if you, if here's the, you know, this is where we can see the problem. Let's say you're like, okay, I'm going to pay you 80 cents and you're going to pay me a dollar. And I'm going to do that because I think I'm really good at flipping coins. And you win on the first one. So I, I pay you the 80 cents. This is, this is what we do in life. You say, see, I told you I was really good at that. That's the problem, right?

So this is the poker problem, right? So if I start moaning, about my bad luck. I'm not focused on the thing that I need to be, which is, am I actually making good decisions that in the long run are going to end up earning me money? And we can substitute money for happiness, wellbeing, fulfillment, health, anything that we value. So we can think about this in the exact same way about like, what are the foods that we choose to eat?

How does it make me feel short run long run, you know, short run versus long run, right? So there's all sorts of stuff that I might do today. That's bad for me in the long run Am I thinking about that and what the effect is gonna be because I don't get immediate I don't necessarily get immediate feedback on that, right? How am I thinking about decisions about what I'm driving who I marry? What job I take, right? These are all in the same category.

And the thing that I love about poker is that it really distills it down into something very understandable that makes it so clear that you have to focus on the skill elements and somehow move yourself past , those luck elements. What I say is like, don't pretend it doesn't exist, which I think is what a lot of people do. They want to pretend the luck doesn't exist. And instead embrace it and just say, of course, there's luck. All I'm trying to do is reduce the probability of a bad outcome.

Aidan McCullen (2): want to really emphasize, thank you for, for bringing us through that. That it's not about poker. This book is not about poker. It's about improving your decision making and dealing. With when the luck doesn't go your way. I was telling you before, Annie, I had a career in professional Ruby. And my son now is really interesting to become a professional soccer player. And he's really interested in, he's only 11.

And he asked me the other day, so innocently, he's like, how did you become a professional player? And I said, in this order, discipline, tiny bit of talent. And luck, and , all you can control is the discipline, like really, like you have talent, but all you can control is the discipline. And I saw so many, I'm sure you did with poker, so many much, much more talented players than me, way more talented, never go anywhere. And then they'd blame the world, they'd blame bad luck or blame.

Oh, you were lucky, you know, you spoke French or something, you know, some crazy thing. And my whole thing was like, walk away, if you're going to walk away or it doesn't work out, at least you'll have no regrets because you controlled the controllables of it all. And I'm saying that just to get us to a key, key point that I really want our audience to hear. We had Ellen Langer on the show a few weeks ago.

Ellen has this beautiful saying, which ties beautifully together with something you said, she said, instead of becoming paralyzed over making the right decision, make the decision and then make the decision right. And it echoed what you say, and this is one of the core things I took from this book, is that there's a huge difference between outcome quality and decision quality. And you need to know the difference with them because it helps you deal with no regrets in life.

Yeah. I mean, obviously a central theme of the book is that there's a huge decision between outcome quality and decision quality. So, you know, I opened the book with this story about Pete Carroll in the 2015 Seattle Seahawks and they're in the super bowl against the Patriots, the New there's 26 seconds left in the game and the Seahawks are on the Patriots one yard line. And there's just a, so there's a clock management problem, which is that the Seahawks only have one time out.

So 26 seconds left in the game. It's actually, there's a constraint in terms of how many plays you're going to be able to get off. It's second down. So they have three downs to try to get into the end zone. Now the expectation of the crowd is that Pete Carroll is going to call for a run play. Specifically he's going to hand it off to a guy named Marshawn Lynch, who is one of the greatest running backs of all time.

And I guess what people think is that Marshawn Lynch is obviously just going to go through the Patriots defense, like a hot knife through butter and score. P. Carroll actually does something really different. He calls for a pass play, into the front corner of the end zone. That ball is intercepted. And the game ends with the Patriots winning. Because the Seahawks at that moment, which I forgot to say, we're down by four. Okay. So down by four, the Seahawks have to score a touchdown.

They can't score a field goal. So they needed to get the six points. So obviously when that ball is intercepted, Patriots are up by four. They can just run the clock out. That's the end of the game. The headlines were just brutal. The fans were just brutal. I still play this video when I give talks and people in the audience, like there's always a reaction. Some people laugh when they see the still that I put up on the screen.

Some people go like, I'll often get like, should have handed it off to Marshawn Lynch. We'll get yelled in the audience. And people just have very strong opinions about this play to this very day. So the first thing that I say to them is, and I'll ask you this, what if it had been caught for the game winning touchdown, what would the headlines have been? Aidan McCullen (2): Master stroke by Carol. Right. Nobody saw it coming. This is why it's gonna be in the Hall of Fame.

So this, this right here should tell you that there's something wrong because the play is the play. You know, this is the problem, like this, is that that poker hand that I went and cried to Eric Seidel about 82% of the time ish, I'm gonna win 18% of the time. I'm gonna lose if I, if I lose the 18%, you shouldn't say I played the hand poorly, and if I win the 82%, you shouldn't say I played it well. It depends on whether I made a good call or not in that situation.

That's really what matters, right? In that particular situation, I think I played the hand quite well, but there's been lots of situations where I thought I played it poorly, right? And this is the Pete Carroll issue, right? Is that if, if that is the game winning touchdown, everybody's talking about how brilliant it is. If the passes. incomplete. People don't make a peep about it.

It's only under these circumstances where the ball is intercepted that people say it was an idiotic play, but the play is the play. The outcome on a single try doesn't tell you anything about whether the play was good or bad. Right? In the same way that if you call a coin heads and it lands heads, it doesn't tell me you're a good coin caller. It doesn't tell me anything.

So what we have to do is get back to, well let's look at what the possible outcomes were and what the probability of those outcomes were to start to try to understand whether that decision was good or bad. And this is like a primary idea in the book is that any decision you make is associated with some set. of possible outcomes. The world is not deterministic. If I make a decision, there isn't only one outcome that could occur. There's a set of possible outcomes that can occur.

And each of those is going to occur at some probability. So my job as a decision maker is trying to say given the probability of any of those outcomes and the payoffs associated with those outcomes, am I making a decision that causes me to advance toward my goals on average? Because I cannot control the outcome that I actually observe. So let's think about the Pete Carroll thing. Alright, so, 26 seconds left, needs to score a touchdown, three possible downs, but only one timeout.

So this is a problem. So I'm going to ask you this question. Let's imagine that he hands it off to Marshawn Lynch, just like everybody wants him to. What happens to the clock if Marshawn Lynch doesn't score? It keeps running. So what does Pete Carroll have to do? He has to use his one timeout, right? Now if he hands it off to Marshawn Lynch again The clock runs if Marshawn Lynch doesn't score, and that's the end of the game.

So he's only going to get two plays if he hands it off to Marshawn Lynch twice in a row on second and third down. Okay, but what if on either second or third down, I don't really care which one, he calls a pass play. Now it becomes interesting because what happens when a pass is incomplete to the, clock? When a pass is incomplete, the clock stops all on its own. So you don't need to use a timeout. So, we can think about three outcomes that are associated with passing.

One is a touchdown, we're obviously very happy with that. One is incomplete. Interestingly enough, we're also fine with that because it stops the clock. And it stops it in about six seconds. Doesn't take very long. And then the other one is an interception. So, let's imagine that we really want to hand the ball off to Marshawn Lynch twice.

If we have a pass play somewhere in there, either on second down or third down, we can hand it off to Marshawn Lynch twice and also try to pass the ball into the end zone. So I assume having played rugby that you would like three chances to score rather than two, right? Okay. so now what we can say is, okay, but like, first of all, What's the chances that Marshawn Lynch is going to score? Because obviously if he's like 90 percent to score, you probably just want to go that way, right?

Well, it turns out he's about 50 percent to score in that situation. So going back to the coin flip problem, what that means is that 25 percent of the time when you hand it off to Marshawn Lynch twice, he will not score. Okay, so let's remember that. He will not score 25 percent of the time. So I think that tells you, okay, let's now look at the other possibility. Close to half the time, that ball will be caught for a touchdown.

Alright, that's pretty good because we know that 50 per that's about how often Marshawn Lynch is going to score on the one try. Close to half the time, the ball is going to be incomplete. So what that means is that the probability of an interception is actually super duper low. Because what you're paying to get that third try, to score is the interception rate. That's the new outcome that can occur. And it turns out it depends on who you ask, but it's one to 2%. That's it.

So we can go back to that poker thing that happened. I lost a hand where he was 18 and a half percent to win and I was like super duper sad, right? Think about how much more that is than one to two percent to win.

So what people didn't understand is that they were just observing an incredibly low probability event and actually the play was brilliant because it gave you three attempts instead of two and it wildly increased the chances that you would win the game because you're basically giving yourself three 50 50 shots instead of two which reduces the chances that you're going to lose the game from 25 to 12 and a half percent cuts it in half so you know it's like

it's clearly just a very good play the problem is that we can't see that Because look at what I just had to do to show you that it was a really good play, right? I had to go through all this stuff and I just say all this math and whatnot. So this is the problem that we have as decision makers is that getting to decision quality and understanding decision quality is actually quite difficult. It's quite hard to know what the quality of the decision was.

And so we use these proxies and the easiest proxy, because this is not complicated at all, Was it a good outcome or was it a bad outcome? Oh, it was an interception. It must've been a crappy call. Oh, it was a touchdown. It must've been the most brilliant call ever. And we use these proxies all the time. And what it does is it really messes up our decision making because it makes people not throw pass palates. in that situation, right?

Or you, you know, it's sort of like the equivalent of like, you go through a red light, you get in an accident and you're like, I'm never going through a red light again. Now, obviously in that particular case, it's like settled. We know, we know that those are the rules of the road, but it's very rare that we actually know for sure whether that was a good decision or not. And we're changing our decisions all the time. So I'll give you like a super simple example.

You hire someone, they work out. You say, I'm really good at hiring. Clearly the process that I use to hire that person is great. Which may have just been like, A friend recommended them. I had lunch with them and I really jived, you know, which it was a very bad hiring process by the way, but like it can work out or you hire someone and they don't work out. And you say, well, obviously I make it made a mistake. I shouldn't hire someone, you know, that fits that profile ever again.

And both of those are huge mistakes, but we make those all the time in hiring because we don't realize like how noisy hiring is. I mean, when you hire someone, it's about 50 50 to work out. If you've got a really good hiring process, you can get it to about 60 40 and that's it. And then we're, we're trying to sort of like change, we change the way that we make decisions going forward based on that good or bad outcome on the one time, on the one hire. That's nuts.

But that's really the problem that decision making under uncertainty is causing. Then there's this other problem that occurs. Which made me think about the Ellen Langer, the Ellen Langer quote rather made me think about it, which is that for a lot of people, we so don't want to make decisions under uncertainty because we're so uncomfortable with it. The idea that somehow we, we can't determine what that outcome is going to be, which is just how life is, that we will go in one of two directions.

I think there's sort of two profiles. One is the sort of, you know, analysis by paralysis, right? That trying to con continually gather more evidence to try to get to a hundred percent certainty before you're willing to make a decision. That obviously is very bad. It really slows you down. It's incredibly bad for innovation. How can you innovate under circumstances where you need to be a hundred percent sure that's actually the opposite of innovation, but this is what happens.

And it particularly happens in settings that, do a lot of resulting. Which is an enterprise setting. And the reason why I know an enterprise setting does a lot of resulting is because they have something called a postmortem and a postmortem is literally resulting. It's like, ah, we had a bad outcome. I'm now going to put you in a room and quiz you about it. Right? Like that's what it is. And when you have a good outcome, there's no quiz. So postmortems are literally institutionalized resulting.

You want to know why your enterprise isn't innovating. Who on earth is going to be willing to innovate because they're going to expose themselves to the probability of a good outcome that's just like way too high. And they're never going to do it. Right? So that's one thing that, so now they're going to get consensus and they're going to gather all this evidence and they're going to come in with a briefing book that they can plop on the table and say, look at all the work that I did.

You can't be mad at me for trying. Terrible. The other thing that happens is I'm just going to go with my gut. So it's almost like a giving up. It's like, well, I can't possibly know. So therefore I'm just going to go with my gut. And both of those are huge mistakes. What you actually want to do is live in the middle where you say sort of like, I'm going to embrace the uncertainty, but I'm also going to figure out what I know. And I'm going to do both of those things at the same time.

And I think that that's actually a better place to live. And it gets to the core of. the title of the book, which is Thinking in Bets, because I think that when you really embrace this idea of thinking in Bets, that gets you into this focus on both knowing what you don't know and knowing what you know. And I'll tell you why. So if you have a gut decision maker, right?

Like someone who's just really into their gut, and they'll declare, You know, something like, you know, we should hire this person. I know they're going to be great. Or we should launch this project product. I know I guarantee you it's going to be amazing because my gut says so. If I say, Oh, really, how much do you want to bet on that? What happens to them every single time? They're like, what? They back off, right?

And they're like, I didn't mean well, no, but you just said you did because you're like, I just know my gut tells me, I have a nose for value. I know what's going to work. And it's like, as soon as you say, but do you want to bet? They get into this more reasonable place of being like, no, what I really mean is like 60 percent of the time it's going to work out. Right. Because that's really what the essence of betting is, is what's the probability and the payoff, right.

Associated with the two things. And am I willing, you know, if I put my resources in here, like how often do I think this is actually going to go in a direction that I like? And that's an amazing place. That's an amazing place to live. And that's what, where I want to get people is away from the need for a hundred percent certainty, which is completely unrealistic. Which you can get to in two ways, because both of them are expressed by a need for a hundred percent certainty.

One is my gut is great. So I'm just going with my gut. And the other is I'm going to do so much work that I never actually do anything. Aidan McCullen (2): And thank you for linking it to innovation because.

One of the things that I want our audience to see our audience or CEOs, decision makers, startup founders, and this is so valuable, this type of thinking, even, you know, getting stopped by Eric Seidel and told, you know, look in the mirror here, chief, that happens to innovators all the time, and it's usually the biggest critic and we walk away from them because they're not confirming the biases we have and actually embracing them is absolutely core.

And I wanted to just say, cause there's a quote by John Maynard Keynes. I think it is that, but worldly wisdom teaches you it's better to fail conventionally than succeed unconventionally. And it speaks to the Pete Carroll call. But one of the things you taught me through this book, and I think it's so important is that.

what you are literally doing with innovation is betting to learn to get feedback the way you would i want to do i want to pull the next card i want you to play an expert so i can build on that if that's the right terminology, what you say that it is often the case that our best choice, doesn't even have to be have a particularly high likelihood of succeeding so even when we play the right call, it doesn't sometimes work and that happens innovation happens in entrepreneurship

all the time and we blame, Ourselves instead of going out sometimes yeah the conditions are wrong the market conditions are wrong it was the wrong time i was too early there was a recession like you say the hot hand fallacy of. So many people who are like great buying and selling property until oh eight hit and then they then they got proved all wrong there's a lot in there but i love you to riff on that. So there's different ways when we're thinking about the coin flip that you might take that.

You can have a positive expectation. Okay. So one is that you could have a coin that is going to land heads a lot of the time. And I'm making, I'm making the right bet for you. Right? So, so let's imagine a situation where the coin is going to land heads 90 percent of the time. Okay. And I'm going to pay you 20 cents if you get it right And you're going to pay me a dollar if you get it wrong.

So this is one kind of mistake that people make where they're like, Oh, I can't take that because I'm only going to win 20 cents when I'm right. But if we do the math right, 90 percent of, 20 cents is 18 cents and 10 percent of a dollar is 10 cents. So you're actually winning 8 cents on that bet. Did you follow it? Was that clear? So basically you're just taking the probability that it lands heads. Cause I assume you're going to call heads. It's a 90 percent point.

The probability it lands heads is 90%. I'm going to give you 20 cents when that happens. So 90 percent of 20 cents. which is just nine times two, right? It's 18 cents. When you lose, when you call heads and that 10 percent hits where it lands tails, you're going to lose a dollar. So it's 10 percent of the time, you're going to lose a dollar, which is 10 cents. So now we can take your winnings, which is 18 cents.

Subtract you're losing 10 cents from that and it's 8 cents every dollar that you bet. So that's an 8 percent return and that's going to be a pretty sure thing because it can be very rare that that coin is actually going to land a dollar. So, so let's call that indexing the market, right? So that, that's sort of what that is, right? Like the market is very likely to go up. You're, you're not really, there's not a lot of risk in this bet. And the return is reflecting that, right?

Because you're, you're getting 8 percent back, which, you know, obviously indexing the market would be pretty good, but but it's a lower return, but you have more certainty about what the outcome is going to look like in the short run. You have more certainty about what the outcome is going to look like. Okay. It can be a mistake though. to go that route. It depends on what your, what your goals are, right?

Like if you're saving for retirement, that's a good idea, but not for example, if you're trying to innovate, right? And this is what a lot of practices and businesses push people to do. They push people more for those sure things that don't have a big return. But now there's another way that you can make a bunch of dough and let's change it to where you're going to lose 90 percent of the time. So basically what I'm saying to you is. You have to, you have no choice but to, call tails.

So it's going to land heads 90 percent of the time, tails 10 percent of the time. I'm forcing you to call tails. But what I'm going to do is I'm going to pay you 20. when it lands tails and you're going to pay me a dollar when it lands heads. So now we can do the same math, right? Like, so we can say, well, 90 percent of the time you're going to lose a dollar. So that's 90 cents. 10 percent of the time, you're going to win 20. So 10 percent times 20 is 2.

So we can take 2, which is now going to be your upside and subtract that 90 cents, which is your downside. Right? And you can see you're making 1. 10. So for every dollar that you bet, you're getting more than double your money back. You're getting 1. 10 back. Obviously that's a good bet. That's really what innovation is, right? So you're going into something that has a low likelihood of succeeding, but a really high payout if it does. Now, there's more risk in that bet.

So, notice in one case, you're making 8 cents. In the other case, you're making 1. 10, right? So, you're getting more than double your money back. In the other case, you're making 8%.

There's less risk in the 8 percent bet in, in, in the sense that you're going to realize your gain in a much shorter period of time, there's much less swing going around in the, in the sort of innovative, in the innovative example, there's a lot of risk involved, meaning that it's going to be a lot of failure, but when you hit that 10%, you're going to make a ton of money. How do we handle those two situations? Well, that has to do with this idea of what are you kind of willing to put at risk?

So you should be willing to bet a lot more of your net worth on the situation where you're going to win 90 percent of the time, not all of it, but a lot more than in the situation where, where you're going to win 10 percent of the time. So when we think about innovators, Basically what we want to do is say, okay, so how can we sort of think about de risking that for someone? And there's a variety of ways you can do that.

One is you could be very young where if you fail, you still have lots and lots of time to do other things. and build wealth, right? So, so just being young is a way to be able to innovate. Another thing is, and this is really what venture capital does, right, is to get venture funding, because what that does is it de risks it for somebody who can't afford to lose, right? And then basically the way that's being paid out is , the venture fund is taking the monetary risk.

And so they're getting paid for that, right? So they're getting equity in exchange for that. And you're putting in obviously sweat equity and you're getting paid for that equity and hopefully everybody comes out of that happy. So that's another way that you could do it. Another thing is you could be rich. It's another thing you could do. You'll see that, for example, with multi time founders, right?

Or somebody was employee number 8 at Uber, obviously they have a lot of money and then they start their own thing, that's a way to de risk. One of the things that I think is really tragic is that enterprises are de risked. They have so much, they have so many resources and yet they don't, they aren't really willing to take these bets. which really goes to the innovators dilemma, right? And that I think is a real shame because they're the most de risked of all.

And yet you really don't see a lot of innovation coming out of coming out of enterprise situations.

I think again, because of a lot of, some of the things that we've already talked about, but I think that's, I think it's really important to understand, and then we can take it a step further and we can say, okay, if the case, if it's the case that Innovation by definition is going to be decision making under the highest degree of uncertainty because you're doing things for which you're cobbling together past experiences to try to figure out what, what might work in the future.

But clearly, if you're innovating, people haven't done it before, right? So this isn't going through a green light. This is the opposite of going through a green light, right? Many people have gone through a green light. We know a lot about it. But this is going into the unknown.

So if we know that we're going to be, that's going to be the situation under which we have the greatest uncertainty, what we want to do is minimize the number of decisions that we're making under conditions of the worst case of uncertainty and start to stage our decisions out in order to start to gather information so that we can improve decision quality as we go along. And the way to do that is really, , that's really what minimum viable product is doing.

It's basically saying , I'm going to do a little thing. So , In the smallest way, it would be, don't build anything. Go talk to customers first, potential customers first. So notice I'm not committing a lot of resources. to go build something if I haven't talked to a customer yet about whether they would be interested in me solving a problem. Is this a problem for you? Is the solution that I'm thinking of something that you would want?

So what that's doing, it's de risking you because it's lowering the resources that you're putting into the decision before you take the next step, right? And then you can start to think about like minimum viable product. And that's what that's, doing for you. Now, again, a lot of time people do the opposite where they build the whole thing and then they're sad that it doesn't work.

And it's , okay, but you put yourself under the stress of the most uncertainty possible when you could have actually pulled that back for yourself a little bit. Aidan McCullen (2): that speaks so much to the idea of going all in on a hand and I thought about the concept of betting the farm we saw this with, I don't know if you ever saw the case study of CNN plus and they, they did it in a vacuum and didn't bet tolearn, but I thought it was a nice segue for a huge section of book.

There's actually two chapters really that are dedicated to this. Perspective taking, how do I get the devil's advocate involved? How do I engage diverse opinions, neurodiversity, et cetera? I thought it was important for us to say a word in that because it really is something you build towards. And this was Eric Seidel talking to you there, saying stuff that is uncomfortable to hear.

And as you say, Do I take the red pill or the blue pill and actually go, I'm going to see the world as it is, because that's actually where the learning is. Yeah. And I want to say, I just, about that, I want to say there's a battle that we have within ourselves between the what we want for ourselves in the future and what we want for ourselves now. So I, I want, I want to eat the cupcake now, but I also want to live to be a hundred, right? And there's tension there.

Do I make myself happy today or do I make myself happy in the long run? And that's really what I was confronted with. And we're all confronted with when we get feedback, like what Eric Seidel gave me, because I could just say, Oh, he's an asshole. And the reason I could say that is because he wasn't making me feel good. But what he was really trying to tell me is, do you care about being a good poker player in the long run?

Because if you care about being a good poker player in the long run, you're going to have to recognize that sometimes you play hands like crap most of the time, actually, sadly, because it's decision making under uncertainty. So you're generally, the default is that you're making mistakes. Okay. So the first thing that you have to do is you have to make a decision that I want to see the world as objectively as possible because that is good for my long, achieving my longterm goals.

It may feel bad in the moment, but I have to be okay with that because this will make me happier in the long run. And I really need to take care of future me because there's a lot more versions of future me than there is. me now. There's infinity of those, right? Basically. And there's just me. But that's actually a really hard decision to make.

So like I really encourage people to try to make that decision that it's okay if you make a mistake now, because one of the things that I think, one of the best ways to get yourself into the situation where you're willing to make that decision is to, I guess, no pun intended, like, or pun intended, have your cupcake and eat it too. So the way that you do that is say, well, and this is kind of what I did.

Remember I told you that you'd be walking the halls of a poker room and people would be like, Oh, that donkey played so bad. And I got so unlucky or, Oh, I played so great and did. And it was always the same thing when they were winning. It was cause they were fricking brilliant and they were losing. It was also cause they were brilliant and the other people were just lucky and they were unlucky. And that kind of horrified me. So I said, Oh, well, I don't want to be like that.

And one of the ways that we really derive a lot of feeling of self worth is through distinctiveness, right? We like to be distinct from other people. This is really a lot of how politics works. It's not just about belongingness to a party. It's about being distinct from the other party. Right. And so distinctiveness is actually a big driver of sort of identity and feeling of self value and really moving along that positive self narrative , that Kahneman talked about.

So , you can, instead of just saying, Oh, I feel bad because someone said a mean thing to me or they gave me negative feedback. If you say, you know what, I'm going to be really open minded to this. I'm exploring with, think about it. You can then say to yourself and I'm doing something that other people aren't willing to do. And that makes me special. So that's having your cake and eating it too.

So that's really where I came to in that was this idea of, okay, I still get to feel good about myself because I'm doing something that these people who I'm listening in the poker halls, in the halls of the casino aren't doing. And that makes me feel good about myself that I'm obsessively trying to figure out what my errors are. So I would really highly encourage that. Now there's a problem though, and there's , two problems.

Problem number one is it's hard to actually get other people's opinions. And the reason is that other people have two issues. One is they're generally going to try to agree with you, particularly if they're your friend. So when you're trying to get their advice they're often just going to give you what they think you want to hear, which is going to be affirmation of whatever choice that you wanted. or they made what I said before, try to conform the outcome to the decision quality.

In other words, Oh, you must've made a bad decision because, or you must've made a good decision because. So you want to avoid both of those problems. So the solution that I came up with, which is one that I advise businesses to do. It's actually how I set up their decision processes really comes down to this really important idea, which is if you want to know what someone thinks, don't tell them what you think and certainly don't tell them the outcome.

So the way you see this in business all the time, so I'll give you the poker example and then the business example in poker, what you'll see is, Oh, I had a hand and someone raised in front of me and I had ace queen and I re raised them and I caught, and they, They called and then the board came in Ace nine three and they checked and I bet and they raised me and I, I called and then the next card they did and all, and then it turned out they had Ace King, right?

So like I, I've literally told you everything that happened to the hand. Not only that, I've told you how it turned out and I've told you all of my own opinions because if I raised with Ace Queen, it means my opinion was I should raise with the Ace Queen. So I've told you everything, right? So now, If I ask your opinion, well, you know, I lost the hand first of all. So you're probably going to look for errors and you're probably not going to want to make me feel bad.

Like, I mean, now I've got all these problems where I'm not actually going to know what you think. So the way that I started describing hands was this, I'd be like, someone raised in front of me, I had ace queen, here's how many chips we had. They'd been raising a lot before that. I w I came across, at that point I was pretty tight cause like I had, I was on a bad streak of cards. I'll give you some details, right? And then I'll say, what do you think I should have done now?

Notice I already did it. I know how the hand turned out. I know what I did, but I'm not telling you. And the reason I'm not telling you is that I want your opinion. So this is the best way to get somebody to tell you you're wrong because they don't know that they're telling me. That I'm wrong. All right. So, so this happens all the time in meetings.

How many times have you been in a meeting where someone gets up and they say, well, I'm thinking about this go to market motion, and here's the strategy that I'm considering. And here's all the reasons why. What do you think? Aidan McCullen (2): Invest in my company. Right? Like you're, you're not, there's no possibility that that team is going to give you the feedback that you're actually looking for. Right?

So, so a lot of what I do in my consulting work is try to make it so that people aren't sharing their opinions with each other before they get everybody else's opinion. So I create like these online forums where people have to go and fill them out before they come into a room and meet with each other in order to sort of recreate that thing that was happening in poker. And the science is actually incredibly strong on this.

That you need to be eliciting people's opinions independently because that's the way that you actually capture the diversity. of the group, the diversity of opinions, the diversity of thought in the group is to allow them to express those independently because otherwise there's too much cross contamination and group think and striving for alignment, which I think is the stupidest word ever and all that stuff.

Okay. So that, that's kind of like the first problem that we have that we really want to address. By just keeping your mouth shut, it's the best way to get people to tell you you're wrong. And you ought to be really happy when that happens, like you're eager to get people to tell you when you're wrong. So you have to set up systems to let them do that because they won't naturally do it.

The second problem is that we're actually really bad in the moment of making a decision in taking into account the future. So this is something that Again, like Daniel Kahneman talks about, he uses the word in it. Hal Hirschfeld who has a great book called Future You. That, that's also, he, he really talks a lot about this problem too, is that we're just not very good at it. You know, we're, we get caught up in the moment and we can't sort of get to that 10, 000 foot view.

So one of the ways to actually get really good perspective. So we want to think about getting the perspectives of other people, but we also want to get the perspective of us in the future. Right. So let's imagine that I'm playing poker and I'm losing and I'm thinking about going and rebuying. And one thing that I know is that when I'm losing, I'm going to be actually a pretty poor judge of why I'm losing. Cause I know that I'm going to have this tendency to blame it on luck.

So if I go and I decide that I'm going to go buy in, I could stop myself and say, When I wake up tomorrow morning or next week, when I look back on this, do I think I'm going to be happy with this decision? So I'm sort of recruiting my future self to now sort of give me some advice here, right? And the answer is going to be, no, you're going to think you are an idiot. And so maybe that's going to stop me from buying it.

So that's one thing you can do is kind of have this interrupt of like, do I think I'm going to feel about this? Like when I look back on this in a week, am I going to be happy? I made this decision. You know, this comes up in like arguments with your spouse a lot. When you're about to say something, it would be good to say like, am I going to regret this tomorrow? Aidan McCullen (2): I love, I love how you put it. Cause this'll help people. It's when Marty McFly meets Marty McFly in the future.

Right. And it's like, don't do that. Stop. Right. Um, so that. Don't kiss your mother. Don't do it. So that, that's one thing that you can do is that interrupt. Another thing you can do is set something up in advance. So what I can do is I can say there's this class of decisions that I think are going to be pretty bad at. And so I'm going to set up what's called a pre commitment contract or Ulysses contract. that's going to actually improve the chances that I stick to it.

Katie Milkman who has a wonderful book called How to Change, has done quite a bit of work on this. David Eagleman is writing a book on this topic as well. And a pre commitment contract is basically, so we can think about the strongest form of a pre commitment contract. Let's imagine that I don't want to eat ultra processed foods, but I know that when I'm hungry and it's late at night and the convenience and whatever, I'm going to eat them. I can throw them all out of my house, right?

So I'm not in the moment of the decision, right? Like I don't want any, I'm not hungry at the moment. And I'm recognizing that when I get hungry because ultra processed foods are quite addictive, I'm probably going to want some. And so what I'm going to do for my future, I'm going to protect my future self and I'm going to throw all this stuff out of the house. Okay, so that, that's one way to sort of take time traveling and bring it to your advantage.

Now, you can't always do something as drastic as that. So what I can do is just actually make a commitment and say, I'm not going to eat ultra processed foods. And then it's really good if I bring you into it and I say, Hey, can you help me? I'm making a commitment not to make ultra processed foods. So now you become like an accountability buddy for me. And that's going to really reduce the chances that, that I do that. So, so that's another form of perspective taking, right?

So there's getting the perspective of other people and how do we actually do that with high fidelity? But there's also getting the perspective of the future version of you, is also going to improve your decision making. . Aidan McCullen (2): I just want to remind our audience that's in quit the idea of kill criteria.

And as you said, your accountability partners had you go, okay, if you fall below, if you lose 600, you're out no matter what, because as you said, in that state, you're probably in fight or flight mode and your decision making is flawed. That was, I thought that was so, so important to talk about and quit. I'll link to that as well. That episode, the book is brilliant as well as thinking in bets. I love this quote, and I thought we really needed to share this because.

The book is ultimately about you having a better life. Yes. It's brilliant for business people. Yes. It's brilliant for decision making in any context. But one of the things I loved was a quote you shared by artists and writer, John Cocteau, and he said, we must believe and look for how else can we explain the success of those we don't like, and it reminded me, I had.

on the show carl taveras on a few weeks back brilliant book mistakes were made but not by me and she talked about how, say if you're having a disagreement over a decision she said in one case psychologist study where they change the author of the paper and then people thought it was a great idea because it was their own, can you say the key is that and explicitly recognizing that we, Field and outcome as a bet, we consider a greater number of alternative causes more seriously

than otherwise we would have had. And that's truth seeking. So this concept would inside an organization and here you share Robert Merton's work where the kudos concept, and we won't have time to go into all that. But I just thought the importance of setting up a, like the, the government has a a line what's the word? Sorry, a dissenting line. So people have the right to dissent. In fact, you're seeking out that dissent.

Like you said, when the customer goes, yeah, not interested in that, I would never buy it. And you go, Oh, that hurts a lot, but thanks for saving me a lot of hassle. I think what you just said is kind of interesting. Cause I think one of the reasons why people don't do that is because they don't want to hear that people don't want it. Right.

Like, you know, it's, it's that thing of like there's, I can't remember who did the study, but, people want to, you know, it's why do people avoid like medical tests? And it's like, well, cause somehow if you don't actually get the test, you don't have it. Because they don't want to get the bad news, but that's actually a real thing. And, and I, I, Aidan McCullen (2): no mirrors anywhere. Right, exactly.

So I do think, I do think that that's, that's, I think there's something very deep in what you just said, right? Like, well, why aren't people just going with a wireframe or something? Right? I didn't quite talk to customers, you know? So I mean, anyway, whatever. I, that just made me think of . Aidan McCullen (2): And it happens in innovation all the time. we fall in love with our idea.

Yeah. You know, the Ikea effect where you're kind of going, Oh, I put so much effort into this thing and in innovation, we do that all the time where, and then to the point of you have a, an Eric Seidel, who's a gift to you who goes, look, I work in regulation. This thing's never going to fly. We're never going to get past regulation. You're going to go, la, la, la, I don't want to hear that type of thing.

So it's, it's this idea of setting up a, a panel or some way for the organization to hear the truth. so, so key. And I thought we'd, we'd end on your advice for that. Maybe work that you do what you're consulting for people to set up the dissent line in an organization. Yeah. So basically the way that I do it in organizations is I set up the decision making process so you don't need a dissent line so we can think about, Solutions that are after the fact solution.

So a dissent line would be an after the fact solution where people can dissent with a particular policy. In this particular case, , maybe they weren't decision makers in the policy and you would need to dissent line. The groups that I work with tend to be pretty small. So. We don't really have that problem, so , we wouldn't have a reason for a dissent line.

But that's what that would be, like in, in the CIA, there's some group of decision makers who do something and then the rest of the CIA, I mean, obviously you're not going to include every CIA agent, right, or State Department person. So, so then you have a dissent line for that. So that's kind of a, the decision has been made now people are dissenting and I think that that's healthy. I, it's not just not something that I have to set up. So that's an after the fact solution.

The other after the fact solution would be you make the decision, you get the outcome and then you go back and do what I call knowledge tracking, which is because I think, I think part of the big problem that we have is we think we knew things at the time of the decision that we didn't know. Right. So when we think about like why are people so critical about Pete Carroll's decision, it's sort of like, they feel like he should have known it was going to be intercepted. Right.

That that's really kind of what the problem is. Like, how could you not know that? And so what's happening is that you're sort of bleeding your state of knowledge today into an evaluation of the state of knowledge at the time. So what I ask people to do is actually write down what did you know at the time, write down what you knew at the time, write down what you learned afterwards. and then figure out, could I have known that beforehand?

So sometimes the answer will be, yes, I could have known it beforehand. Then you have to ask what the cost was. Cause you know, you need to make a decision in a week and it's going to take you a year to know it. Doesn't matter if you could have known it beforehand before you made the decision, cause you didn't have time. If you could have known it within the timeframe that you needed to make the decision, then that's fine. You just add it to the decision process going forward.

But mostly the answer is going to be no, I couldn't have known it beforehand because mostly what you learn is the outcome, which you can't know beforehand. So those are kind of after the fact solutions. What I do with the organizations that I work with is I really try to come up with before the fact solutions. So the, before the fact solution would be I'll give you an example from, first round capital. So I'm a special partner at first round capital.

And what we do there is, we have a point of view, which is expressed in a decision rubric about what are the things that you should be judging that would tell you whether you should invest in something. So you can think about broadly market founder team product. Okay. So let's think about broadly those four categories.

And then within each of those categories, we have what we call mediating judgments, which are things that you would have to judge in order to be able to judge the founder well, or things that you would have to judge in order to be able to judge the market. Well, so you could imagine rating something about Tam. And that would tell you something about the market, right? About the market size. Is it growing? Is it shrinking? Those kinds of questions. So we have this rubric for each thing.

They rate, the favorability of those things on a scale of one to seven. So notice we're sort of quantifying qualitative opinions here. Okay. And then when they make a rating of, say, the market as a whole, they give a rationale for why they're giving that rating. Alright, so here's how we sort of avoid a dissent channel. A founder comes in and pitches. That looks normal. It's a normal pitch. The founder leaves. None of the partners say anything to each other.

Remember that thing about keep your mouth shut, right? So none of them say anything to each other. They go to an online forum, which is this rubric where they fill out all these judgments on a scale of one to seven. And then they give some rationales and there's some forecasts and so on and so forth. That all gets dumped into a spreadsheet. So now we know where they disagree with each other because it's already been recorded.

So we have a pre recording now of what the point of view of everybody was right at the, at the time. And now they have a discussion about those opinions. So we're not actually soliciting anybody's opinions. in the group discussion because we already know them beforehand. This is really important. So then you lose that cross contamination and all of that stuff. And now what's driving the discussion is the places where people disagree. So the discussion will kind of go like this.

Everybody thinks the market is pretty good, but there's a lot of disagreement about whether this is the right founder. And that's what you talk about. So literally it's an acknowledgement that people are pretty. much in agreement about the quality of the market. They like the market that that person is playing in, but they have a lot of disagreements about whether this is the right founder to invest in. And that's where we're like, why do you rate them high? Why do you rate them low?

Why are you sort of meh? You know, and we're sort of driving discussion around where the dispersion of opinion is and then they make a decision. Okay. So notice now, the dissent is already built in, right? Like what we've done is we've created a decision process that gets the dissent to show up. And this now makes it a lot easier to do a look back of the decision because I know we've all been in situations where someone was like, well, I knew it was going to turn out this way.

And I totally disagreed at the time. And you're like, I was in that meeting. No, you didn't. And this is like this memory creep that occurs with hindsight bias, but now we can just go look. And we can say, okay, we invested in this company. It did really poorly. What was our point of view at the time? Sometimes you can say we invested in this company. It actually ended up raising a really strong series A or a really strong series B. What was our point of view at the time? Did we miss something?

Right. And now we can actually go look. And it just, it makes the decision process so much better. It makes it so much easier to close feedback loops. And it's a thing that I could never do in poker. Because in poker, it's this real time, I can't go off and be like, Hey, I'd like to get everybody's opinions here. Could you write them all down before I decide what my move is going to be? So it actually solves a really big problem that like poker players have. And the science is very, very strong.

is a place where the science is really good. That , this way of making decisions really improves decision quality. And then you don't need a dissent channel Aidan McCullen (2): ,Brilliant. And one of the things I just wanted to really nail was that John Cocteau quote while I shared that was, if I'm a, if I'm a company, Like your company that you work with there.

And I look at it, somebody else's, if I'm not trained in your way of thinking, I kind of go, those idiots, I can't believe they made those mistakes. And what you're trying to say is put yourself in their shoes.

So have empathy, actually make the decision, have an assumption, I captured that assumption and then check against what actually happened in that as well, which is such an important thing in, in all aspects of life, because we do it in like that fundamental attribution bias We go, those people made such a terrible decision, but when it happens to you, it was luck all the time. And you're trying right?

Exactly. And I think, I think the other thing, I think the other really important thing to understand is that let's say that I'm, that I'm playing against someone who's really bad. It doesn't mean that they're worse at everything than me. So I ought to be paying attention to them and try to figure out what it is that they do well. Cause I need to know that, first of all, I need to know that in order to play well against them.

But I also need to know that cause I might be able to incorporate that one thing into my game. So when you're looking at another company and you're, you're like, what a bunch of idiots, they're morons. You should be like, but what are, what are they doing better than us? Like you should actually be super obsessed with that. And I think that people are the opposite of obsessed with that. I think that it's, it's so, you know, it's so nice to eat the cupcake and just be like, I'm better than them.

Woohoo. Right. And, and boy, that, that just is really bad for improving and learning. Learning. Right. Aidan McCullen (2): And that outcome bias you talked about, , when most of us in business school, we'd learn by case studies, but we know the outcome, that's the worst. Aidan McCullen (2): we know the outcome. It's crazy. Yeah. And, and, you know, you're, you're hypothetical. What would you do if these were your chips? This was the time, these are the cards.

What would you do is a much better question, but very hard to. Bring into the education system at this, at this late stage, last one for you. This, this dawned on me, right? So I recently, I was thinking about, there's a guy I know I had on the show. He's retiring from a major consultancy. And I thought, I wonder what would happen if that consultancy interviewed him and created a digital twin of all his knowledge, and then do they own that IP? Does he own that IP? Can they use it again?

Do they hire other people, et cetera? And I thought it about you. I thought myself, I wonder, is there going to be an Annie Duke bot, right? Which should be the decision making thinking in bets, decision making applied to business, what would you do? Because, and then I thought to your point about chess, it would be easy to do for chess, which is why we've seen the AI is created that beat Lee Sedol in the game of go, we've seen deep mind chess, Gary Kasparov, et cetera, et cetera.

And I wondered, Can AI play poker given all the different permutations that happen in poker versus chess being a very mathematical game? Yeah. So AI has gotten pretty good at poker, actually. And a lot of it has to do with poker can sort of be boiled down to, game theory optimal. So this goes back to John Nash and a Nash equilibrium table. so You know, there is some art to it because you have to, you have to have some sense of what the ranges of hands that your opponent is playing.

But basically you can generate a table for any situation that tells you what you should do with this range of hands, this range of hands, this range of hands in any situation. So, you know, clearly you have to be smart enough to memorize that. But I would argue that an LLM is going to be better at memorizing those tables than I am. And, but, but it took a long time, right? So Deep Blue was in the nineties.

They started getting pretty good at poker kind of the mid 2015, maybe ish, somewhere in there, where they started getting, you know, being able to beat people heads up. And now I think they're probably pretty good in, in at least low stakes. Poker games. So there's a some, you know, but, but again, there, there's, there's a game theory optimal solution to any situation that you're playing in, in poker.

You know, I retired in 2012. So if I tried to go back and play now, I would get slaughtered unless I was willing to do the work of, of. really memorizing those, those game theory optimal tables. People hadn't really generated them back then. We under, we understood that idea, right? Like we had some knowledge of it, but you couldn't do the kind of data work that you can do now.

And now it's kind of insane what people have available to them in terms of these, in terms of these solutions and what we call poker solver. And because I retired in 2012, I haven't studied the poker solvers. And so I'm sure I could beat you at poker, but I would not try to beat a professional at poker anymore. I would, that would not be good. In terms of, you know, what I do, you know, I'm sure at some point there'll be, something that can do what I do. That's fine with me.

Aidan McCullen (2): You created it though. Yeah, I guess. Aidan McCullen (2): your point about Pete Carroll. Pete Carroll probably, they probably use real time AI to give permutations of the next play. well, that's the thing is that they have, they actually, so, so I think, I'm not sure that they're allowed to do that, but because they, the NFL makes you use paper. But I know that they, that's sort of what those binders are, right?

It's like they have these binders and there's, there's someone sitting there sort of looking up the situation that they're in. Pete Carroll knew that he was making a good decision there. He's, he's defended it. By the way, other people have defended Bill Belichick defended it. I think generally within the, you know, sort of like the analytics people in poker, the people who are more analytics minded, they all understand that he made a good decision. I mean, it's not like he got fired.

The fans were mad. The pundits were mad, but the people who really knew were like, yeah, I mean, he made a good decision, but you know, this is also a bind. Right. And it's just sort of speaking to this because of innovation. One of the things that I think is really interesting about the Pete Carroll situation is that if he had handed the ball to Marshawn Lynch twice, which would have been a very bad decision. And that's all he'd done and he had lost the game. Nobody would have cared.

And the reason that it was, it was expected. And so there's this interesting thing that happens, which is if you do sort of the expected thing, the not innovative thing, the thing, the thing that's like going through a green light. Nobody gets mad at you when you fail. Nobody, nobody, because they're like, well, what could you do? You went through a green light. And it goes back to that postmortem problem, right?

Which is, okay, so now you're going to make me gather way more information than I need in order to decide, because I want to defend myself in the room where I'm going to make sure that I've consensus across the whole company, or I'm going to make sure it's just SOP. Cause that's going to get me out of it. It was the innovation. It was that it was unexpected that made it so that that outcome got such a big response. So when you do something that's SOP and you win to it, people are like, good job.

And when you lose to it, people are like, yeah, bad luck. When you do something that's innovative and you went to it, they call you a genius, which is what would have happened had that play worked out. But they also call you like an idiot and a moron if you lose to it. So this becomes a real dilemma, right? So in a startup, the expectation is failure. So you can innovate all you want, right? Because there isn't, there's no SOP, right? Everything is innovation. It's just the expectation.

But once you get into an enterprise setting, that's not the case anymore. And now all of a sudden it's like, it's very hard for people to do new things because the critique that comes. from failing in those things is so huge. So yeah, like for Pete Carroll, the insiders know it was a good decision, but look how much flack he takes. 10 years later, people are still like, what an idiot. Aidan McCullen (2): Great point, and one we'll finish on.

There was one line I wanted to share as a finale, and it was you talking about yourself, but I think we can all derive a lesson from it. You said you rewired your reward systems and habits to get your dopamine fix. By trying to be the best credit giver, the best mistake admitter, and the best finder of mistakes in good outcomes. And I thought that was what you could do in businesses, what you can do in sports, what you can do, what you can teach to your children.

To go, that's all you could do actually. If you could, if you do those things, you'll make progress and you'll inch forward and you'll win the flips over a period of 10 years as they come towards you. always a massive pleasure having you on the show. Annie, for people who do want to find out, I know you have a sub stack channel, you have courses now in Thinking in Bets. Where is the best place for people to find you? yeah. So I run cohorts on maven. com if people want to take a class for me.

The other thing you could do is show up at Wharton for executive ed, which I do twice a year with a whole host of other amazing professors like Murray Schweitzer Joe Simmons, Adi Weiner, Kate Massey. I mean, it's really like, it's an amazing group. If you, if you want to go take an exec ed cast at Wharton, you know, check that out. I have my substack, which is called Thinking in Bets. And then, you know, the easiest place to get in touch with me is through AnnieDuke. com.

Aidan McCullen (2): author of Thinking in Bets, Quit, and How to Decide, Annie Duke. Thank you for joining us. Thank you so much for having me.