Barry Ritholtz's Masters in Business: Philip Tetlock Interview

of Pennsylvania. Professor Tatlock is really a fascinating guy. I first got to know of his work through a book he wrote well over a decade UH ago called Expert Political Judgment, and it really cast an enormous shadow of doubt on all of these self proclaimed experts and and he used the field of political science as his basic investigation arena, but really it applied to everything from investing to economics to anywhere people prognosticate about the future and

make um confident sounding assertions about what's going to happen one to five ten years in the future. He pretty much destroyed that entire line of thinking and showed that the average so called expert is no better than the average person walking down the street. That was well over a decade ago, and he tells an absolutely fascinating story of how we're familiar with DARPA, nette the Defense Research

Projects invented the Internet and all these other really cool things. Well, the intelligence community has their own version of DARPA and it's called IARPA. And they basically approached Professor Tetlock and said, we're fascinated by your work and we'd like to find out if there are a way to to either identify or develop those outlawyers in your studies who actually turned

out to be pretty good forecasters. And as it turns out, forecasting is a humble skill that can be I don't want to say learned by just about anybody, but you can undertake a number of steps to make your own forecasting better than it might have been otherwise. And it's fairly rational and fairly straightforward and really really fascinating. And so you'll hear the story of of not only how the original book came about, but how the new book

came about and how he modified the previous view. It turns out, looking twelve months and further out, it's still pretty much a crapshoot. No one knows what's going to happen. But on shorter periods of time and on very very specific things, you can undertake a number of steps that will help your own ability to discern probable outcomes be much better. I think if you're at all a statistics

or probability want you're gonna find this absolutely fascinating. Anyone who's an investor, a trader, who who deals with market or economic forecast might also find this to be really really interesting. So, without any further ado, my conversation with Professor Philip Tetlock. This is Masters in Business with Barry

Ridholts on Bloomberg Radio. This week on Masters in Business on Bloomberg Radio, I have a special guest, Philip Tetlock, professor at the University of Pennsylvania, where he is cross appointed at both the Wharton School and the School of Arts and Sciences. Perhaps best known as the author of Expert Political Judgment, How Good Is It and How Can We Know? Which has won a number of awards. We'll

talk about that in a little while. His most recent book, also winning accolades, is called super Forecasting, The Art and Science of Prediction. The Economist magazine named it to one of its best book lists of Professor Tatlock is also a co principal investigator of the Good Judgment Project, which is a multi year study looking at improving the accuracy of probability judgments of real world high stake events. Professor tat Luck, Welcome to Bloomberg. Oh, thank you, great to

be here. So I'm familiar with you from your your earlier work, some of your earlier books, which I found absolutely fascinating. But let's just start this segment talking very very generally. How come there's such an enormous appetite for political and economic punditry, including all the predictions and forecasts that go with that. Well, it would be deeply dissonant for people to um to come to the conclusion that

it doesn't matter who's elected president. It doesn't matter whether we raise taxes or cut them, whether we help the Ukraine or don't help the Ukraine, or intervene in Syria, or don't intervene in Syria, or assign free trade packs or don't do that. Um, we could just toss a coin, because nobody the is on either side of the political spectrum has any demonstrated ability to assign good probabilities to the consequences of those policy options. Uh So it's to

say that sounds nihilistic, it's dissonant. Um people just don't want to live that way. People don't want to live that They want structure. And for the same reason that people have turned to psychics and which doctors and all sorts of things over the past. We we we we we we see guidance, predictive guidance, and we seek it. There's a demand for it even when the supply of

good forecasting is extremely limited. So we're recording this in the month of March not too long ago, the cover of Barons magazine has a picture of Hillary Clinton and Donald Trump, which one is better for investors, and the usually conservative Barons grudgingly says, we think Hillary will be better for the stock market then than that Donald will. Don't we run the risk of putting way too much credit for good markets and good economies on the president

and vice versa. When things are bad. Don't we blame them too much for what's gone wrong. I think there's a great deal of superstitious reasoning about the impact that that that leaders have on organizational performance and also the impact that presidents have on on the economy. The number of leaders that presidents have to influence economic outcomes is quite limited um, but there is this deep intuition we have that leaders should be accountable for what happens on

their watch, like the captain of the ship. It really doesn't matter whether it's a storm suddenly emerges out of nowhere. Uh. You you, you want to hold somebody accountable and and that that is also deeply wired into us as it's how someone has to be accountable for it. So, so let's ask the obvious question, why are experts so often so wrong in their forecasts. Well, there are a couple

of theories about that. One is that the problem lies in the experts and how the experts think, uh, and they could do better if they had used better analytical tools and we're more self critical and creative and thoughtful. And the other theory is that we just live in a radically unpredictable world in which originally nobody can do appreciably better than chance over extended periods of time. It's a world royaled by black Swanish dark gray Swanish events

uh and uh, com buyer beware. Fair enough, So let's talk about good judgment in predicting future events, or at least good analytical processes. And along that that line, I would be remiss if I didn't ask what is the Briar score and why is it so important the Briar score. Well, the Briar score was originally developed by some statisticians UH working with meteorologists who wanted to keep score. And this goes all the way back to and it's a very

simple idea. UH. You want to minimize the gaps between probability and reality. So your code reality is either zero or one, depending on whether the event occurred or didn't occur, and you have probability judgments that range from zero to one, and you you get a really good Brier score if you assign probabilities very close to zero to things that don't happen, and probability is very close to one point

zero two things that do happen. So it's your ability to be justifiably decisive to make appropriately extreme judgments as the circumstances dictate, but to avoid driving off a cliff in the pursuit of that objective by making extremely overconfident judgments and saying ntent of things that don't happen and of things that do well. Once you're thinking in terms of probability, aren't you already light years ahead of the folks who are that decisive in making these bolds outlawyer declarations.

But isn't that much more nuanced than thoughtful? Then here's what's gonna happen A And of course there's a big possibility that it just doesn't go that way. That's right. The probability scale and principle, if you want to look at it where a mathematician would is infinitely divisible, an infinite number of points between zero and one point zero.

And obviously people can't distinguish that many levels of uncertainty. Now, when IBM S. Watson was playing in the Jeopardy competition and beat the best human players, uh, you might have noticed that under its answers, occasionally there would be this little uh Baysian probability estimate of how confident Watson was in his answer at point eight seven three six two or something. Uh So, these these types of form forms of artificial intelligence do try to uh make extremely granular

distinctions among degrees of uncertainty. Human beings can't make that many uh distinctions among degrees of uncertainty in most environments. I mean, there's some environments where we can pull out a calculator and do it. If you can do it, with card tricks and so forth um or poker um.

But there are there are real limits on how granular you can become when it comes to whether there's going to be a country leaving the Eurozone in the next year, or whether there's going to be a violent Sino Japanese class in the East China Sea, or things of that sort. How many degrees can you distinguish their Is it just yes or no? Or that yes maybe no? Or can you make finer distinctions than that? I'm Barry Ridhults. You're

listening to Masters in Business on Bloomberg Radio. My special guest today is Professor Philip Tetlock of the University of Pennsylvania. He is the author of a number of books, but the one I want to talk about right now was the award winning Expert Political Judgment, How good is it and how can we know? Uh? This won a number of awards, the Graumeyer Award for Ideas Improving Political Order, the Woodrow Wilson Foundation Award for Political Science, and the

Robert E. Lan Award for a Political Psychology. Let's let's start with a quote of yours that I want to get some some feedback on people who make predictions in their business who appear as experts on TV get quoted in newspaper articles advised, governments and businesses are no better than the rest of us at making forecasts. How is that possible? It turns out that you reached the point of diminishing marginal returns for knowledge quite quickly and a

lot of the domains we care the most about. Now what does that mean? UM? When I started off doing research on political judgment, UM, one of the greatest psychologists and on the planet advised me, Daniel Kahneman and the Nobel Prize award winning psychologists most recent book, Thinking Fast Right.

And it was a lunchtime conversation about thirty years ago in which he said rather casually that he thought that the experts I was interviewing from my early work on expert political judgment would have a hard time doing better than UM an attentive reader of the New York Times. UM, which is, you know, a kind of a fancy way of saying more or less what you just said. Uh. Now, he he didn't know that as a fact. He was offering that as an hypothesis. Uh. And I think the

right way to look at this is, UM. It is an hypothesis we can be continually testing. It's it's not always going to be the case that experts are going to fall short, but they're going to fall short much more often than we would expect. Um well, how much of that is random? And when they're right and at a certain point don't don't the either investing or let's call it, voting public have a reasonable belief that the supposed experts know what they're talking about. When they make

a forecast. They expect them to be considerably better than I think as as someone once called it a dart throwing monkey. Right, Well, there's a there's the big question that we want the answer to, and then they're all these proxy cues that we kind of latch onto in the hope that those proxy cues will get us closer

to the answer. So the big question we want the answer to say is whether the U. S. Economy is going into her session next year, whether that the Dow is going to be over twenty thousand or under fifteen thousand.

There there's some big questions that people in the financial community or political community want answers to, And there are various people who passed through our lives passed through your radio station, passed there in the op ed pages of newspapers, on television and so forth, who offer opinions on these things, and they come with various types of credentials. You might say, so and so is the muckety muck professor a bl bludy blum, or you might say that so and so

when a Nobel prize. You might say that so and so is worth ten billion dollars, or you might do a lot of things you could say about that and um. The The interesting question is, uh, do those things give us much guidance on how accurate what the person is saying? So we're hoping that they do. We're we're hoping to say, well, this person must know what he or she is talking about by virtue of the fact that this person has done X, Y or z. But the relationship between having

done X, Y or Z and accuracy is unknown. And the more honest we are about our ignorance, the more honest we are about when we're using proxy cues for judging how credible source of advice is, the better off we're going to be in the long term. And and by the way that applies to me too, that's fascinating. So so let me ask, um, when you put out expert political judgment, had anyone really done a full on quantitative analysis of how accurate experts were, at least in

the political field. Had had anyone tried to figure out, Hey, let's figure out exactly how right or wrong these folks are? Before? Interestingly, very little work had been done. There was a little bit of work assessing the accurate Well, there's quite a bit of work assessing the work of weather forecasters. There was some work assessing the accuracy of expert bridge players, uh, And there was some work assessing the accuracy of economist.

The Federal Reserve in Philadelphia and elsewhere had been doing some some studies along those lines. But as for assessing the accuracy of political pundits, at the time my book came out, I think there was extremely little work on that subject. So here's a quote from the book, and I want to get some um feedback from you on this. When they're wrong, they're rarely held accountable, and they rarely admitted.

They insist they were just off on timing, or blindsided by an improbable events, or almost right or wrong for the right reasons. Well, if you're a pundit, You're playing a complicated game. Uh. If I'm a pundit on your show or anyone show, and I need I need to make it sound as though I know what I'm talking about. I need to make it sound as though I'm telling the listeners something they didn't know before. I also need to preserve my long term credibility, which means I have

to have some escape clauses. So if the claims I make about the future turn out to be wrong, I need I need some way about of walking away from it. So when you have this problem of um, you have a career as a pundit, you need to be saying something surprising, but you also need to preserve your long term credibility. That's a real dilemma the pundit is in. So the typical way pundits cope with this is by saying something dramatic, uh, like Canada will disintegrate or the

Eurozone will disintegrate, or Pudin will reinvade the Ukraine. But build in some waffle words like this could happen, or this might happen, or there's a distinct possibility this will happen. Now distinct possibility is one of those wonderful phrases, because if it happens, I can say, hey, I told you there was a distinct possibility. If it doesn't happen, I can say, hey, I just that was possible, So I'm

covered either way. Uh, And that UM, it helps to explain why pundits um and and indeed why traditionally people in the US intelligence community as well UM have relied so heavily on vague verbiage forecasting because they need to be saying something that sounds informative, but they need uh strategy for preserving their long term credibility at the same time.

So that explains why they don't admit error. Although I would argue nobody bats a thousand, admitting error shows that you have a little humility and recognize that it's not easy. In the last minute we have in this segment, the real question I have is why don't we hold these folks accountable. Well, here's the thing, Barry, they don't even

think that they're wrong. If I if I say there was a distinct possibility that Putin is going to invade Estonia this this coming year, and he doesn't do it, I'm gonna interpret distinct possibilities having met a very low probability, and if he does do it, I'm going to interpret distinct possibilities having med a very high probability. I'm gonna we we tend to be somewhat self serving in our own mental calculus. I say, well, you know, we we interpret,

we give ourselves a benefit of the doubt. On how we interpret distinct possibility adjusted, we give the benefit of the doubt. You know political pundits who favor our political party. I'm Barry rid Helts. You're listening to Masters in Business on Bloomberg Radio. My special guest today is Professor of Philip Tetlock. He is the author of Expert Political Judgment, How Good Is It? As well as Super Forecasters, a

new book that just came out to great acclaim. He teaches at the Universe City of Pennsylvania, Wharton And and let's jump right in to the Hedgehogs versus Fox's discussion. You referenced this throughout really throughout um the second book a lot, and if I recall correctly, you mentioned it a few times in the first book, which I've read a while ago. Uh. For those people who may not have read Isaiah Berlin's essay explain to us what is

the hedgehog and the fox? So Isaiah Berlin was British um scholar, political philosopher, political historian, philosopher who um took a quote from the Greek warrior poet or Kilicus from years ago and he built a really interesting essay around it and the quote. One of the few surviving fragments of this man's work was that the fox knows many things, but the hedgehog knows one big thing. And he intended

that to capture different styles of things. King he thought that um, some some thinkers were much closer to being um uh foxes. Shakespeare, I think was one of his classic examples of a fox who could just a very multifaceted view of human nature, and other writers um he thought could be could be pigeonholed better as hedgehogs. Now we use this fox hedgehog distinction um in the work on political Judgment because it rather captures rather well uh

different styles of thinking. Um. You could be, for example, a hedgehog of very many different political sorts. You could be a free market hedgehog, or you could be a Marxist hedgehog. You could be an environmentalist, uh, doomster hedgehog, or you could be a boomster um utopian kind of you techno utopian sort of hedgehog. You're going to find a cost effective substitutes for whatever we're running out of.

So there there are many different forms of hedgehog left right, pessimistic, optimistic, and we identified many of them in the early work, and we tracked their their their accuracy, and we compared their accuracy to that of more fox like forecasters, and we found a couple of things. One is that the hedgehogs tend to have pretty bad batting average when you look at all their predictions, uh, their their overall batting

average is pretty bad. We also found that the hedgehogs tended to be more prominent, They're more attractive to the media. The media like the kinds of sound bites that hedgehogs can deliver. And we found that the hedge hugs um

also tended to have at least a few home runs. Matt, if you're making a lot of pretty extreme predictions on a wide range of subjects, at least a few of them are almost by chance going to be accurate, whereas the foxes are more making more moderate probability judgments, and they have less claim on home runs. So um, you get on somewhat ironic situation that the worst forecasters have

the greatest media prominence. Isn't that inherent to the process of not only having a real specific expertise in one area as opposed to being a generalist, but also making those outlier forecasts. I use a slide in my presentation about a particular pundit who every year for the past seven years has forecast a seven like crash every year, and you would think the media would eventually say, hey,

this guy is just consistently wrong. But it's such an outrageous forecast and it gets people so excited they love to bring them back on. Isn't that the nature of sensationalism that the hedgehogs are going to be more, especially today where everything is on clicks and views. Who's going to generate more clicks a rational sober well, we don't really know what's gonna happen, versus the sky is falling

and everybody loves that. Well, I would just suggest that that people be better off if they were more honest about the functions that are served by consuming different types of information. So you if you said to yourself, look, I want to be entertained. I want I want to see somebody who's saying outrageous things and I'm gonna be But but I'm not gonna base my probability judgments on them.

I just going to want to be entertained by these amazing stories as person is going to tell about how the Saudi regime is going to disintegrate and how we're on the verge of World War three. Uh, this is this is really entertaining stuff as opposed to listening to this much more at tentative, nuanced fox like forecaster who's saying on the wine hand, on the other hand, drones on and on. Why there's really only about probability of the Saudi regime changing in the next twenty four months.

Can we really view the world through the lens of a of a single defining idea or is that, as the disclaimer says, for entertainment purposes only. Well, we need to be very clear about the functions that are being served. Uh. Some of these big ideas are very useful lenses for viewing the world um at particular moments in history and in conjunction with other ideas. So I'm not saying that

the intellectual apparatus is useless. I'm saying that it's what's really dangerous is when you have a smart person who runs too far with a big idea and fails to see that the complexity of the world puts a lot of breaks on it. So one of our rules of thumb for distinguishing better forecasters and worse forecasters on the media is the ratio of the number of times they say however versus moreover. So if you have a high however over moreover ratio, that means you're a fox. That

means you're boring. That means are probably going to be kicked off the show. And if you have more more accurate, more likely to be more accurate. But that's right, you're gonna have better briar score. I'm Barry Ridholts. You're listening to Masters in Business on Bloomberg Radio. My special guest this week, professor of phil Tetlock of the University of Pennsylvania. His most recent book, super Forecasting, The Art and Science

of Prediction. So let's jump right into this because I have so many questions about this, and let me start out by just asking how many piano tuners are there Chicago, UH somewhere between about eight and I think so. So it's a fascinating question because your initial reaction is to shrug and say, I don't know. The variation I've heard of that is how many cardiac surgeons are there in London? UM or what's the empire state building way? That's another

good question. So so let's talk. Let's talk a little bit about what do most people do when presented with a question like that? Interesting? You know, some of the high tech firms in Silicon Valley were quite fon fond of asking off off the out of left field questions of this sort because they thought it was a great

way of testing how well people think on their feet. UM. In the book, we call these Ferremi questions, named after the great Italian American physicist and Rico Ferremi, who developed the first um UH nuclear reactors keep part of the Manhattan Project developing the bomb, and Ferremy was fond of

UH posing these oddball questions to his students. He what do he wanted to the students to do was to take an seemingly intractable problem and break it down into parts or components that were more tractable, So you might not have any idea how many. No one has any idea initially on how many how many piano tuners that might be in Chicago. But you make guestiments about the population of Chicago. Walk us through that because you you go through about seven steps and you get pretty close

to the correct answer. Right. Well, you're you're making a lot of guests and it's not just the breaking down and get trying to get the answer. What you're doing in the process is you're revealing sources of ignorance, and your colleagues on your team, for example, if it's because our forecasters often work together on teams, uh, your colleagues

can help you correct help correct your errors. So what's the population of Chicago, I don't really know, between two point five and four million, and maybe you know some somewhere in the middle there, what proportion of the population

would have a piano and so forth? You see, if you would, you would break it down and you you you try to mind eventually how how how many people could conceivably make a living working as as piano tuners given the number of people who own pianos in Chicago and their willingness to pay for the services of piano

tuners um and so so Faremy did this. He One legend is that he tried to uh infer the strength of the first atomic blast um by dropping little pieces of paper when the when, when the when the winds came in front, and by estimating how far the winds blew. And I think he was off by about forty or fifty percent. But you know that for Faremy estimates, that's not too bad. It's it's a lot better than simply shrugging your shoulders and saying I have no idea. I

say a ten Kelton glass as post of twenty. But that's that's fascinating. I'm absolutely entranced by Firm's paradox, which says, where where is everybody? You know, it's a giant universe filled with different galaxies and hundreds of billions of stars? And are we really the only intelligent life here? And I found most of the various arguments both ways to be lacking. It's really it's really a fascinating, fascinating debate.

But let's let's stick with this. So so looking at what the empire state building ways, or or how many piano tuners in Chicago show us how to break down unknown questions into component parts and make reasonable assessments and reasonable valuations on each of those segments. Um. So, so what do you find about teams of super forecasters? How much better are they at at these sort of predictions

than the average person, or prediction markets or just regular pundits. Well, Um, the teams have super forecasters truly astonished us because the statisticians were telling us the right thing to do here was to have each individual top performer make judgments completely independ at lay of the others, um, rather than allowing

them to contaminate each other. And you get conformity, and you get group think, and you got to get kind of a blur rather than a number of distinct points of view, and then you can combine them statistically somehow. So Um, we were the only competitor in the forecasting tournament sponsored by the U S Intelligence community that used teams um And but we we hedged our bets. We weren't sure that teams would work. We ran an experiment and we randomly assigned people to teams, and we random

and had other people work as individuals. Uh. And we were truly surprised that the teams functioned as well as they did UM, and it's an interesting question of why our teams were so dynamic and open minded relative to many teams you see in actual organizations. But before you answer that question, let's let's put a little flesh on the bone with some numbers. So teams of ordinary forecasters

beat the wisdom of the crowd by about ten. They were bested by prediction markets UM and addiction markets beat ordinary teams by about and then the super teams of the best forecasters beat the prediction markets by anywhere from fifteen. So these folks working in groups really are the outliers. None of the other groups are even close to them in terms of of accuracy. Why do you think that is, Well, yeah,

and I'll just add one other thing to that. They weren't just outperforming prediction markets in the public sphere, they are also outperforming intelligence analysts who were working, you know, behind a veil of of of classified information. And it's just a remarkable thing. And I think US intelligence community, which is much maligned for many things, deserves some credit for its willingness to sponsor of forecasting tournament that has

the potential to be embarrassing for government bureaucracy. How often do you see a government bureaucracy, uh, spend money, a lot of money on a project that has the potential to be to yield results that you're fairly embarrassing. How hard is it to cultivate these skills or is it just a matter of internalizing these ten bullet points? I wouldn't say just a matter. It's it's no, it's an it's a non trivial thing to do. Uh it's it's it's pretty hard. Um. And uh so let's let's take

get an example of one. And what do you think is an important um, an important commandment of super forecasting? You want to just pick up one at random? Sure? Okay, Um, well, one of them has to do with granularity and uh it's it's it's actually grounded in a story that about President Obama and how are he reacted to his advisors who were um offering him um somewhat conflicting probabilities on how likely it was that Osama bin Laden was residing

in a particular compound in Abadabad, Pakistan. As we all know now, he was indeed residing there, and the President did authorize a n a vcal mission, and that resulted in Osama bin Lad's death. UM. Now, when the President was confronted by these probability estimates from really smart people at the top of the intelligence apparatus, ranging from about you know, thirty five or up to about um the

President's reaction was an interesting one. If if you Hee had computed the average of the median estimate of the advice he was getting, he would have said, looks like about a s probability. But instead he said something interesting. He said, well, look, look, guys, this is a coin flip. It's a fifty fifty thing. Um. Now, the President, I think is a very intelligent person, and I think he's capable of being very granular in his assessments of uncertainty.

And if you doubt it, think about the following thought experiment, which is appropriate given it approaching March Madness. He follows March Madness he uh, and basketball. He's a basketball fan. Imagine he'd been sitting around with friends waiting for Duke to play some team in March Matt Martin the March Madness tournament, and uh, they offered him exactly the same probabilities about whether Duke would win the win the game.

You know, somewhere between thirty five and ninety five, with the center of gravity of opinion around seventy, would you have said it sounds like a fifty fifty thing or would you have said, MM sounds like about UM three to one, Duke UM, I think to ask the question is to answer it. He would have seen opportunity for being much more granular in making bets about sports than he would and making estimates about the likelihood of a

particular terrorist being in a particular location. UM. Now it turns out that UM for many categories of problems where we think it's impossible to be more granular, it is possible. And that's one of the things super forecasters have learned that there's a difference between fifty and sometimes they can even make distinctions between fifty and now we quote that the Chief Risk Officer of UM a q r M, the hedge fund UMAST is the head of that, and

the chief risk Officer is Aaron Brown. And when we talked with with Aaron, he you know, he he's also a really serious poker player, UM, and he said, well, he can tell it different. World class poker player and a talented amateur on the basis that the world class player UM knows the difference, and then he paused that maybe it's more like UM or indeed two, how granular can you get in poker? Well, poker is a game

with repeated play, quick clear feedback. It's possible to get more granular on poker than it is about the location of terrorists or about whether countries are going to leave the Eurozone. But it's an open question of how granular you can get UM. And you need to grapple with this distinction between precision and pseudo precision UM. And that's

one of the things. Super forecasters are just very thoughtful people who pushed the frontiers of knowledge as far as they can, and that means sometimes pushing them a little too far, in which case they retreat. If people want to find your work, just google Philip Tetlock and they'll be able to dig up all of your various publications, books, writings, etcetera. Yeah, Google scholars probably a little faster, but all right. If you've enjoyed this conversation, be sure and check out all

eighty three or so of our prior conversations. Be sure and follow me on Twitter at rid Halts and check out my daily column on Bloomberg View dot com. I'm Barry Ridhults. You're listening to Masters in Business on Bloomberg Radio. Welcome to the podcast. This is Barry Ridhills, Professor Tetlock. If I don't remember to say this later, thank you so much for being so generous with your time. This

is really um been a fascinating conversation. I have a lot of things to go over with you in the last twenty minutes or so we have, but there are a few questions that I'm just dying to ask you because it's your lat previous book really was very influential

to me on expert political judgment. It was that book, and it was a prior book called The Fortune Sellers that really was more of a media criticism of this parade of people who would come through the studios make their outland just forecast, be completely wrong, never be held accountable, and then they would get called back and the more outlandish, uh, the better there were. There was an author who wrote a book I'm trying to remember what year the book was.

It was called The Tao Jones T. A. O. Bennett Goodspeed, and he called these folks the articulate incompetence plural, meaning that they're very good salespeople. They can speak, but really they have no expert knowledge. And if you've spent any time in green rooms in various television studios, there's something to that. So let me ask the basic question that we kind of skirted around during the broadcast portion. Why are we so enamored with forecasting and forecasters despite their

terrible track records. Well, I think because there's a lot of motivated reasoning going on. As we noted earlier, there's this tendency to use a lot of a do a lot of a verbiage forecasting, to to paint a dramatic scenario and then hold it together with some very weak verbs like this might happen or could happen or the

distinct possibility of this happening. Um. So, there's this interesting tendency that the pundits have of engaging our attention with a vivid scenario disintegration of the Saudi regime or you know, uh, Cino Japanese wars, something Tom Clancy issue on that kind of scale, um and and and but to stitch it all together with terms weasel weasel word terms that allow them to retreat later on. And UM, we don't distinguish

very clearly in our own minds. We we don't think we want to hold as they say, all of the fault lies with the pundits. They couldn't do this unless unless we were willing partners. UM. And I think that you know here here I am talking to radio station one of the most influential companies in the world, Bloomberg. UM.

Bloomberg is a major purchaser of expertise. UM. Bloomberg could actually change the world to some degree if it implemented systematic uh, if it implemented systems for tracking the accuracy of many of the people who came through. If part of the price for getting onto Bloomberg was that you had to demonstrate that you were engaging in some kind of rigorous scorekeeping, and Bloomberg could flash up some of

some batting average statistics. UM, as you as you appear, Um you Bloomberg could increase the collective IQ of our society. It could increase the collective IQ of the conversation. Um. When most pundits stay away, Well, well that's the question I mean. I I say Bloomberg because Bloomberg is so influential, I think a lot of a lot of punets would say, well, I'm not I'm not going to run away and high from blue Burg. But the other media could do this

to the Wall Street Journal in New York Times. There are lots of major media science that have the leverage that could induce pundits to be much more intellectually honest. They choose not to exercise that option. Um probably because they don't perceive a great market demand for it. We were talking during the break about my usage of a little app called follow up then dot com. Whenever I see an outregeous forecast, I just shoot an email to a specific date, so Gold going to five thousand dollars,

and I send the forecast. I send the email out to Oh, well, let's let's let's give them a year, so we'll send a forecast out an email out March one at follow up then dot com with the headline and the web address of the article that made this forecast. And then a year later, or if I write March one, five years later, comes the email back that specifically gives me that lank. Oh it's a reminder. Here's what this

person said a year ago. And occasionally I get to do an article about, Hey, here's a wild forecast that someone made and it's been completely wrong. So let's talk about your book. You kind of call me out for calling someone else out, and I'm curious as to your

perspective on this. So in two thousand and ten, when the FED was in the midst of doing quantitative easing, uh, there was a letter published and I believe online and it ran in the Wall Street Journal in a number of places warning that quantitative easing was going to cause hyper inflation and collapse of the dollar and all these terrible things. And so I figured, you've got to give those people three years. So I set a reminder for three years later. And three years later it popped up. Hey,

the dollars at multi year highs. There there is no hyper inflation, there's there's deflation. These guys were wrong, and so I called them out about it, and it went totally viral, got picked up by a dozen different media outlets, and a book called super Forecasting. Now at by today we're six years forward, um, and we still have a strong dollar and and no inflation. What is the issue

with an ambiguous forecast? With no specific I described forecasts as an asset class, a price, and a specific date. If you leave out the specific date, do you get to say we're never wrong because there's just wait, you'll see. Is is that a fair defense of that? It's definitely not fair, but it it is the state of the

art at the moment. You know. There's an old communism joke that that that's rather a proposed here um the Soviet revolutionarily on Trotsky after he was thrown out of the Soviet Union by Stalin, when it went around giving talks to the left wing audiences around the world, and um, one probably apocryphal story has it that he went once when he was introduced to an audience of followers. Uh, the speaker said it was was proclaiming Leon trotskya visionary

who could see far, far into the future. And you're saying, and you know, comrades, the ultimate proof of the far sightedness of of of of comrade Trotsky, not one of his predictions has yet come true. That's how far sighted he is. The the old joke about market forecasting is you could give a price level or a date, but

never both at once. And that's just another way. So so how long do we allow a forecast to persist before we say, all right, at this point it's been X number of years, you we're gonna have to put you in the incorrect column. Well, there's there's no absolute rule, because that's that's the way vague verbiage is vague. Verbiage is vague because it's it's just it's it's always a slippery, vague um thing that no nobody knows how to quantify

it and and keeps it, keeps the pundit safe. Um. I mean, I have lots of forecasters from the earlier work who predicted that Canada would disintegrate, or Nigeria would disintegrate, or there are a lot of disintegration scenarios out there that haven't happened yet. Um. And if you call them on it, are they going to say, well, we're it's it's only seen it could still happen, absolutely and and

wholeheartedly believe it. That's right. So how much of this is just simply humans not being immune to human behavior? You're a psychologist, Let's let's let's take that tact. Is this just ordinary human behavior refusal to accept responsibility for error, not wanting to admit being wrong, not wanting to do anything that reduces their potential um status within the hierarchy. Is that all this is well, we're moving into world that's requiring us to make ever subtler distinctions among degrees

of uncertainty. This orts of distinction as we didn't have to make in our evolutionary past um when we were wandering the savannah plains of Africa. You know there were are there is a lion or isn't a lions during in the long grass and you're gonna you're gonna make you have to make a judgment call pretty darn quickly. And if you dontle very long, and maybe you're not likely to pass your genes onto the next generation. You're better off being wrong but jumping the gun then having

a higher, better track record. But if you're wrong once, well then it's catastrophic in terms of progeny. And that makes a lot of sense. That sort of thinking is why we have a tendency to do all sorts of things that just are inappropriate in investing but worked really well way back when that's right and um, if you can imagine a scenario, here's the here's the big problem with tail risks and scenarios. Um, most of the time people underrate them, but as soon as the scenario has

called to their attention, they overrated. Well, it's very very caul for people to strike the right balance in dealing with tailor risk scenario. So so that whole recency effect thing is since nobody, very few people were forecasting the sort of financial crisis we had in O eight oh nine, and since then it's been nothing but catastrophic forecasts from the recession never ended, We're going to turn into a depression.

Here comes another eight seven like stock crash, the parade of horribles just have not now auto loans and the new subprime it's going to be just like, uh, is this just that that tel risk factor is so recent in people's minds and they missed it coming, and so now they're just like every general fights the last war. These folks are still fighting the previous financial crisis and the diplomats try to avoid the last war. Um, yeah,

I think that's right. I mean, you are you you you go from Iraq to Syrian Libya or you that you go from one error to another. And Iraq was far enough after the Aetnam that it looked like all it takes as a generation before those lessons are lost, more or less yet more or less. All right, so I only have you for another ten minutes, and um, my last question before I jumped to my my standard questions.

Burton Malkiel said, when investors moved from stock to stock or mutual fund to funds as if they were selecting and discarding cards in a game of jin rummie, what does this tell us about human's ability to participate in uncertain equity markets? Well, this is another one of these the ten commandments that we formulated from observing the super forecasters.

They're acutely aware of the principle of error balancing. If you look at the research literature on human judgment, there are two kinds of errors people can make in that situation you're describing. One of them is the error of excess of volatility, of um jumping every time there's a little bit of news and exaggerating its diagnostic value visa

deep market trends. And the other big mistake you can make is excessive rigidity and being so committed to a particular preconception about where the future is going that you just ignore the news altogether. And you don't you fit, you fit, you failed to do any updating. So it's it's it's kind of principal error balance things. But like

writing learning how to ride a bicycle. Um, I mean, I could talk for hours about the principles of error balancing and everybody is like, yeah, yeah, I kind of get at Sure you can make one air, you can make the other. But the only way it's really going to sink into people's heads if they is if they go to forecasting tournaments and they actually practice making judgments, get on the bike and try to ride it. Uh.

And that's what going to forecasting tournaments all about. That that's why we were continuing to run the gj open dot com, which is a forecasting tournament where people can indeed uh work to cultivate their skills. That sounds pretty pretty fascinating. So that's that's really interesting for that. Actually, we have a little more time, so I'm going to

keep bangalway on some of these questions. In the first book on in the previous book on Expert Political Judgment, you know, I look at these is really two sides to the same coin. The first book talks about what is essentially long term forecasts really a year and further out, and they tend to be wrong. The book on super

forecasting is really looking at a year or less. So are are they really saying two different things where we're really looking at two different types of forecast, two different lengths. I think that's a superb point. There are different time periods in the different studies. UM. I'm much more optimistic that we can improve forecasting using the right selection, training, teaming tools in shorter time periods up to about a year.

I become progressively more pessimistic when you go out to the longer reaches of three, five, ten years that were included in Expert Political Judgment, and there I think it's it's going to be very, very difficult to do much better, UM, do much better than chance. How many times do you have to shuffle a deck of cards, um, until it's perfectly random? That's a good question. Well, I think the UM I would guess five. The statistician and magician Percy

dot Com is Stanford. I think he estimated at seven. Well, life is like shuffling the cards? How how many months, how many years have to go by before so many random contingencies accumulate that no no human being could conceivably have anticipated anything that far out. And um O our current best guests for the kinds of geopolitical geoeconomic questions we're looking at, it is around a year or so.

Isn't that just the nature of society and a complex system such as fill in the blank, stock markets, economy, geopolitics, elections, anything along those lines. They're so sensitive to an all conditions, they're nonlinear that you end up with a little change here has out sized impact further down the road. Really, what we're saying is the universe is pretty random beyond twelve months. Uh, it's practically impossible to make any sort

of realistic forecast with any degree of specificity. Um about certain things. I made a bet four years ago with um I won't mention his name, but he's been a guest on the show as to I was wondering who was the GOP nominee likely to be? This is literally three years ago after after the election, and he said, well, what about the Democrats? And I said, well, it's easy, that'll be Hillary, but who I have no idea who the Republican is going to be. So we made a bet,

and so far looking pretty good. Um. But I don't know if I got lucky, just got lucky and assumed it was she was up next, she was next in line. Uh. Under normal circumstances, when you're looking out to three or four years, there are so many contingencies. Can anybody really I got lucky. I'm not pretending I have any expertise in politics or anything else, But can anybody consistently have any sort of acumen thinking out more than twelve months?

It's it's really hard. I mean, there are there are some categories of questions where great longer foresight is possible. I think the Hillary thing was in the cards for for for for quite a long time. Um. But for most of these things, I mean, who mean to take some extreme examples, I mean, in nineteen forty Dwight Eisenhower was an anonymous Army colonel. In nineteen fifty two years president United States in twelve years, right, uh in um Um.

Jimmy Carter was an anonymous peanut farmer in the nineteen sixty four. In nineteen seventy six, he was being elected president United States. Twelve years is a huge amount of time in politics, So I don't think anybody really is going to suppose there's very much possibility there UM, But as you get closer and closer, it gets more and more possible. That's not all that surprising. It's an analogy.

Will be like to Snell and I char when you visit your optometrist and engage, it's easier and easier the closer up you get for most things. UM. The trouble is we're just not very well tuned to the parameters UM. And if somebody can tell a really good story about a relatively far off future about you know, the United States is moving toward a techno utopia and which DDP will will skyrocket as intelligent machines do amazing things for

us in the fourth and Dulsta Revolution. That that meant may that scenario may indeed materialized by its UM, but the likelihood of scenarios of that sort being accurate UM is extremely low. I have a t shirt at home. It says Where's my jet pack? I was promised the jet pack by the year two thousand under forty four characters. Right. That's right. When when you look back at future forecasts from decades ago, and we now have enough for them that we can look back years as to what people

were expecting from the future. What's fascinating is all the amazing technology, technological developments, all the advantages of hardware, software, biotechnology, medicine that we practically take for granted. They weren't the things that people were forecasting. It was colonizing Mars and

other sort of hoverboards and other such things. Uh So, even when you're thinking in terms of giant technological changes, and of course there's a handful of people, you know, Arthur C. Clarke is notorious for having forecast everything from cell phones to satellites to to what have you. Um what does this say about our ability to understand the few,

the present and extra appelate to the future. What it suggests is we'd be better off if we were aware of our limitations, achieved a certain baseline of appropriate humility, and got in the habit of keeping score, and resisted being sucked into clever scenarios and storytellers, and resisted being seduced by credentials. If we could manage to do those things, I think we would um proceed through life, making investments

and political decisions with better calibrated probabilities. And I think we would be better office individuals and we'd be better officers of society. That sounds that sounds tremendous. On a related note to that, because because those those seven bullet points are very significant. Let's let's talk about uncertainty, which you is a is a uh concept that is dotted throughout actually both books. Um, what is uncertainty and what does it mean for individuals just trying to navigate their

way through the world. Do we understand uncertainty? Uh? Do we misunderstand it? What? What exactly is it relative to thinking about the future? Well, um, there are some types of problems where the probabilities can be readily computed. We can compute the probability of drawing an asis spades from a randomly shuffled deck, um very accurately, uh, too many decimal points if we want whatever one out of over fifty two works out too. We can do that with

coin toss games and so forth. Um. So there are some games in which the classic rules of statistics want oh one very clearly apply and there are well defined probabilities. In other words, we know what the range of outcomes are. We just don't know what the specific outcome is going to be. Got a well defined sampling universe. You've got clear, quick, clear feedback about your about your predictions. Um, my of the world, most of the world isn't like that. It's

not like it's definitely not like that. And and and the question is what are the limits? How useful is it to apply probabilistic forms of reasoning um outside their traditional domains of application. Then, in a sense, is what the U. S. Intelligence community really wanted to explore? I mean, can we do better than say, distinct possibility, which, when you look at it carefully, is such an elastic term.

It could mean anything from one percent to nine. So let's talk a little bit about the intelligence community and the Defense Department. Um, how did you get involved with DARPA and the the competition, the forecasting competition? Right? Well, Um, it was I r as DARA, the Intelligence Advanced Research Projects Agency is post to DARPA, but it's it's it's a cousin and and it and it models itself to

some degree. I think after dark by it it really wants to do radical earth change, world changing forms of research. And I think changing how we think about uncertainty would would be would be pretty fundamental. Maybe not as fundamentals inventing the Internet, but way up there. Um, it would be a big deal, um to to to change how we how we go about doing things. I think our our democracy would would be transformed. I think the finance industry would be transformed. It would not it would not

be not a small thing. These would not be small things that how long have they been running this contest? So they started. They approached my wife and May when we were still on the faculty at University California, Berkeley about six years ago, and we had a nice um um a set of drinks over at the Clermont Hotel in Berkeley, and um we um. We were just astonished that the U. S Intelligence community was prepared to run a series of forecasting tournaments. I mean, I predicted that

they would never want to do anything like that. So it's kind of ironic, right that I I forecast that forecasting tournaments would be impossible. And I know it's being a bit of a HEDGEHOWK. What I what I said is, look, government bureaucracies don't give a slingshot money to David right

Glass doesn't give sl slingshot money to David. Why would be a massive influential government you're oocracy fifty billion dollars or so, I wanted to spread millions of dollars around to a bunch of small scale academic competition to see whether or not they can do a better job of assigning realistic probabilities to things of national security significance. And this it didn't make any sense given the normal rules of bureocratic behavior in Washington, d c UM and so

I was too Hedgehogy, I was wrong about that. Now when they I'm delighted. I was wrong to say, at least when when they came to you the prior book, the Expert Political Judgment book, you had really run a form of this. You would assembled a mast over eighty two thousand separate forecasts from several thousand, was it or several hundred political forecasters? It was it was a smaller number. It was in the hundreds, but um the um. Yes.

In a sense, Expert Political Judgment was a small scale dry run for what I RPA did on Expert Political Judgment was run more on a shoestring budget, whereas UH the r PA forecasting tournaments were run on on a much more in a much more per fessional, large scale basis with and you know, and one of the nice things about the R project, and people worry about the applicability of research and things like this, but this was

all independently monitored by the U S intelligence community. I mean, these these forecasts were submitted at nine am Eastern time every day on the day, every DA, every day. The forecasting tournaments are running over four years UM, so there is a very clear paper trail. So what what is the state of the forecasting contests these days? Is it something that they've put aside? What what is the takeaway

from all that? The takeaways are that it is possible to make better probability estimates of events that many people thought it would be impossible to estimate probabilistically. And it's possible to do that by engaging in systematic talent spawning,

which you can only do if you're tracking score. And it's also possible to do by designing good training modules, by putting together teams that are open to dissent and uh know how to do precision questioning of each other's assumptions, and also by doing a little bit of algory with mcmagic um. So do you think the result of that contest has changed the way the US intelligence community recruits talent,

trains talent, and makes forecasts about future events. Well, you'd have to ask the U S intelligence community about how exactly things have changed. My understanding is that the National Intelligence Council now does try to quantify it's probability estimates rather than using just vague verbiage forecasting UM. It has probability ranges. I think it tries to distinguish at least seven degrees of uncertainty, which is a lot more than three. H is more than five, which was which was the

preceding number. I think they may be underestimating themselves. I think they could probably get up to ten or fifteen if they if they wanted to. Uh. But I think they're moving in the right direction. I think there's growing interest in crowdsourcing forecast There's growing recognition that UM, the average forecast derived from a group of forecasters, is often more accurate than most of the individuals from whom the average was derived. It sounds kind of magical, but it

makes it it is true. UM not always true, but it's a good way to bet we We've I've been critical of some of the prediction markets, not because the theory underlying them is wrong. But very often they're narrow, they're not diverse, they're not incentivized. All the various things you need for a prediction market to work is often missing. Um And sometimes the better as the participants are are so similar to each other, it's hard to extrapolate that

out to other other factors. Um. Uh, These these sort of contests and the various prediction markets. Can we describe these as moneyball for the intelligence community? Is it just quantifying data in a way that hasn't been done previously to intelligence forecasts. I think that's a great way to describe it. Just money ballfing the intelligence community. Um. I think it's the movie The The Old World was a world with baseball scouts. Uh, cl Clint Eastwood, Trusty baseball

scouts too. You know, we're gradually being displaced by these number crunchers. Um. We're never going to do away with people who have deep qualitative insights into the subject matter. There are crucial source of inputs. But the question is what roles should we be playing as the world changes? And I think human judgment will always be playing a critical role when we're dealing with human beings. Um Um.

But there are useful tools for combining human judgment, and you can get more out of it than previously supposed. That makes a lot of sense. But before we get to our our favorite standard questions, anything from super forecasting, I might have missed that you want to uh add as as worth thinking about before we uh we get into a little bit of your history. Well, the thing that I most hope if I mean I'm getting older now, I mean I've been doing this stuff for thirty plus years, UM.

And the thing that I most hope lasting legacy of this work, and I hope it improves US foreign policy and intelligence analysis, but I also hope it improves our democracy. And I think in the in the closing chapter, we talk about the debate between Paul Krugman and Nil Ferguson on various issues, and how it more resembles a food fight than it does a serious debate between extremely intelligent people, which which both of them obviously are. UM. And the

question is, could we use forecasting tournaments? Could we structure them in ways uh to facilitate more civilized debates on issues that matter. So that's why I wrote a piece in The New York Times several months ago with Peter Skoblick on how we could do that with Iranian nuclear deal? Um, and when a one way to proceed would be to say, Okay, you've got hawks, you've got doves. You have different opinions about what the long term consequences of signing this deal are.

We don't know for sure which historical trajectory were on. Why don't the hawks generate five questions that they think they have a comparative advantage in answering? Why don't the doves generate five questions and they think they have a comparative advantage in answering? And you know what, victory will have a clear cut meaning here if the if the doves can answer the dove questions better than the hawks, and they can answer the hawk questions better, then the

doves win and vice versa for the hawks. Uh now, um, anyone take you up on that? Well, we do have a number of people who are participating in the tournament, and um, one of the people, and g j open dot com has written a memo on on where where we are right now? The moderate seemed to be doing the best at the moment, but you know that game is far from over. I mean, this is just very early stages of a long term process. Yeah, where you're one of what a tenure treaty, it's quite a way

is to go. So so when you when you describe that, and you mentioned debates, I immediately thought of the political debates this year, which at least on the GEOP side, have been not your usual policy debates. UM. And I'd love to see some of the tenants from super forecasting find its way to uh, the political parties and and see if we can have a little more substantive discussion about when this happens, here's what happens in the future,

and then hold these folks accountable. That really doesn't seem to happen on whether with political experts or politicians. We really don't hold their feet to the fire much do How far into the future might it be when in a in a presidential election, the presidential candidates take pride in what their briar scores are. UM, I think it's a long way off, judging by what's going on this year,

to say the least. So So let's talk about UM, let's talk a little bit about you personally, rather than than some of the books and the ideas that you've you've put forth which which have been absolutely fascinating. Um, So how did you find your way? You went, you became a you went to Yale, you got your pH d in psychology? How did you find your way into forecasting? This really seems far afield from the traditional UM academic realm of that. I was always a pretty strange psychologist.

I I had interest that it took me pretty deep into social science, into political science in particular, into into areas of business. But they always interested in organization as I was interested in societies and cultures and large entities that were not you know that obviously psychology matters there, but it's it's a stretch for a psychologist. So in my early work I did do a fair amount of experimental UM work, but I was also also did a lot of archival and naturalistic work. So it was a

kind of a natural progression for me. Um, who are your early mentors? Uh? Well? Um uh. Peter Suitfeld was my very first mentor in Canada. I was an undergraduate University of British Columbia, and and he was wonderfully supportive and of me, and and he believed in me, and he he really told me that you know I would probably have a pretty good time if I went to graduate school at Yale and I took it on faith. And I did that, and I met a number of

people at Yale who helped me. Um. The guy who coined the term group think, Irving Janice, was one of the people I worked with, and he was quite an unusual psychologist. Also. UM. When I got to Berkeley, of course, UM Daniel Koneman came along in a few years, and he certainly had an influence on me. I already had a PhD, and I was I was just recently tenured faculty. But k Koneman is as a lot of very influential guy. He's he's just a lot smarter than than most of us.

So it's a it's a good idea to listen very carefully when he speaks. I really enjoyed, UM, thinking fast and thinking slow, the metaphor for that entire two stage way to look at how humans make decisions, either fast and instinctual or longer and thoughtful, really just seems to make a lot of sense. UM. What other books, uh, have you really enjoyed? Whether books have been especially influential to you. Another person who influenced me is just um

uptown here at Columbia University. Robert Jervis his book Perception, Misperception, International Politics. It came out when I was in graduate school and I could feel myself being tugged towards these topics. It was. It's a brilliant analysis of mistakes that have caused unnecessary wars, that perception and misperception in international politics. By anything else stands out is as interesting or unusual to you. Well, um, I mean life evolves in funny, quirky,

path dependent ways. I mean, you can you can look back on your life and you can say, well, it was kind of inevitable this happened or that happened. But a lot of the things that that led to my early forecasting tournament work where I think kind of quirky, I mean, it was really kind of quirky. Thats. A scholar as young as I was was appointed to a National Research Council committee when I was just thirty or

thirty one year four. Uh, when I when I was that young, um by far the most junior member a committee like that, and had a lot of senior scientists on it. But it gave me opportunities to meet a lot of people, and it connected me to resources that made it possible to do the early forecasting tournament work. It also impressed on me the need to do it because there there we were five liberals and conservatives, all had very strong opinions about the Soviet Union and where

things were going. And the liberals thought that Reagan was sending us to where the nuclear apocalypse, and the conservatives thought that then the Soviet Union wasn't ere as an evil empire would never change from within, essentially, as you just had to keep up endless pressure and maybe it would eventually crack. But you know, they didn't help a lot any and they certainly didn't see garbage Of as

much of a change agent. Initially, um each side, neither side really predicted gorbachalv and what Garbagechov did inside the Soviet Union in the internal transformations that occurred. Both sides could readily explain after the fact what happened. UM. So it was this mismatch between virtually zero predictab ability and virtually perfect expost explanatory ability that troubled me. And I thought, well, you know, if if debates this important, like World War three,

are are being conducted, this shodily. You know, surely there's a better way to do this um And that's what led to the early work on expert political judgment. It was a way to try, what can we do to to keep score and and if we do keep score, can we identify um better ways of making judgments. You describe something that is an enormous pet peeve of mine. In the markets, nobody knows what happens day to day.

There is zero predictive analysis, and then on any given day, the market's up five hundred points, it's down five hundred points, and ex post there is always a fantastic narrative explaining exactly, here's why oil shot up and why the market rallied three hundred points, or here's why this terrible thing happened and the market dropped five hundred points. But nobody is saying,

if this happens tomorrow, then here's a result. It's always an after the fact narrative that that seems to be consistent across lots of different uh fields, not just politics, but markets and economics. And after the fact we're fantastic storytellers. Before the fact, we have no idea, And there there are situations where we really want to continue doing that, though it's not always bad. I mean, the National Transportation Safety Board, for example, conducts these ex post postmortems on

plane plane crashes. One of the reasons why air travel has become as safe as it is is because they're so good at doing these postmortems. Obviously, they can't predict which planes are going to go down, but they become really pretty adept at identifying the critical factors that underlie plane accidents, and as a result, the rules for pilots and the design features of aircraft have changed in ways

that make us all safer. Safer so um. But they're not just making up a story for the six o'clock news. They're saying, hey, you know the whole shuttle investigation, the O ring fair, that's right. Therefore, that's the most most youth worlds. Most sectors don't have a black box that say, hey, here's why the engine failed at two oh seven and fifteen.

That's right. So there are different types of postmortems, and some of them are constrained by well defined bodies of scientific knowledge and investigative procedure that reduced the serious risk of capitalizing on chance, and others are just sort of make it up as you go, and I think the things we're talking about and make it up as you go. But there are approaches to doing case studies and learning from the past that are very disciplined and focused and

can make a safer and wealthier and happier. The The recent book Um the Checklist talks about how much better surgical procedures and outcomes have been since surgeons started using checklists, including wash your hands, which very often was just assumed um that it was done properly with a certain disinfectant

in a certain life of time. But we've we've apparently dramatically reduced operating instruments left in abdomens and keeping count of the number of sponges, and that's improved the subsequent outcome, just as the National Transportation Safety Board has improved the safety level of of travel. That's right. So the big question for us is when is learning possible? When can we learn to do certain categories of things better? And when are we just spinning our wheels and deluding ourselves?

Are we spinning our wheels and deluding ourselves about financial markets? So many political issues, but we're actually making real progress in the domains of medicine, or airline safety or or whatnot. And it's a it's a mixed picture. Um. And I suppose what we're trying to do with this forecasting tournament work is to bring some of the rigor that has worked in these more scientific domains to bring it to bear in in domains that they are more or less

like the wild West. Huh. Um. So let's you've been doing this now for you said, thirty years. What's changed in this industry more than anything? What is the significant progress in in the forecasting and prediction industry? Um, during the course of your career. That's a hard question. Um, They're not all there there there, that's right, well there there. I I think that our knowledge of the imperfections and human judgment, thanks to a lot of the comment inspired

research programs, I think we've made discernible progress there. To the least, I think we've I think some of the statistical tools have improved in various ways. I think some of the tools for running teams have even improved. I mean, I think we can do There are versions of the Delphi procedure, for example, which was developed a long time ago, but has got the DELFI better, right, so, which is what I remember. I was saying that a lot of people thought it was kind of crazy to use teams.

You're better off having a lot of independent observers. But there's a way to get the benefits of independence and the benefits of creative interaction at the same time. And one way to do that is by going getting everybody to make their judgments anonymously. So you give your probability judgment, your explanation, I give mine, and everybody around the table gives theirs, and we circulate, and we we circulate that and nobody knows who said what. So the high status

guy isn't swaying everything the way off. It often happens in groups. UM, and everybody's expressing your judgments anonymously is don't, so they're insulated from the group thing pressure. And you can do that two or three times, and then the question is how much better is the resulting group judgment after you go through this process. Uh, Then it would have been if you'd simply say, taken um an unweighted average of each of the individual group group group judgments,

and the answer is this better? How much better? Uh? I think met analysis suggests probably in the vicinity of ten percent better is you know? That's real? Right? And if if you're talking about avoiding a war or finding a terrorist or anything along those lines, that's a real worthwhile pursuit. It's nothing to sniff at, nothing to sniff at. So now let me ask you for your forecast. What are the next major changes? What are the next shifts that are going to come in in the world of

forecasts and predictions? What do you see? Perhaps the better way to say this to ask this is what do you see as the influence of your work on on the forecast and community? I see huge potential here, um i R, which funded the first forecasting tournament, is going to be funding to follow up forecasting tournaments, um and I think many of your readers might be interested in these, and there will be calls for volunteers to participate in

one form or another. One of them is focusing not so much on the accuracy of your forecast as on the probitive value of the explanations you generate. The probate of value. Are you good at for or after the fact? Are you good at explaining things? Um and? And again, the extent to which we can crowdsource aspects of problem solving and then eventually marrying that. I think the forecasting, UM, I think that's a very ambitious project. It's it's it's

in the very early stages. It hasn't been launched yet, um, but I'm optimistic that it will. UM. Um. So what this is called the I r PA forecast things. It's called create create c R E A T E. Uh. It's it's it stands for the flex reasoning. That's right, it's an acronym like that. All right, that sounds interesting. I'll definitely I'll search for that and all link to

that and that. The other is another competition which is called h FC, the Hybrid Forecasting Competition, which will be UM, humans and machines and human machine combinations, uh, trying to make predictions, which I am quite optimistic about. Um. So it's Watson working with somebody, well, it would be that would be one way of doing it. There are a lot of there are a lot of possible machines, are a lot of possible models that people could work with.

And the question is are you better off just using the model or you better off just using the person, or you're better when are you better off using the combination of the model and the person, and the one truth is we really don't know the answers to these questions right now, and we're hoping to learn more. So I think that's these are these are really important projects.

And I think the other big thing that that I'm very focused on because it has relevance to improving the debate in our debates in our society, is competitions to generate better questions. It's not to generate better questions. It's not just about forecasting. It's I mean, you can you can forecast trivial pursuits, and you can become a great forecaster in trivial pursuits. And really, the world is not a better place for it? Or what? What? What? Your world will be a better place when we join super

forecasting skills. Two questions on which big policy debates pivot. So you say, if we knew the answer to this question, would we have invaded Iraq? Or if we had known the answer, if we had known the answer that if we if we knew the answer this question, what would would have changed how we what we do in Syria, the Ukraine, or with respect to tax policy, with respect to uh FED policy or whatnot uh so uh generating

probitive questions, generating high quality explanations, human machine competitions. I think these are three really important areas for the future. That's fascinating. The last two questions, Um, this is always interesting and I'm trying to figure out the best way to its phraser for you. Normally, I say, what advice would you give to a millennial or someone just graduating college who are going into your field? But I don't know whether that's the field of psychology or the field

of analyzing and improving forecasts and predictions. But let me ask, in an open ended fashion, what advice would you give to someone just coming out of school starting their career who wants to follow in your footsteps. Well, it's an interesting point that I really don't have a field anymore. I mean, the University of Pennsylvania when they hired me,

they didn't really know where to put me. So I'm partly warden, partly psychology, partly a political science, and Nannenburg um so a lot of different But it's really the you're really studying this, let's let's just, for lack of a better phrase, you're studying the science of decision making that we're studying human judgment and the extent to which human judgment can be improved using a variety of tools, some of them drawn from psychology, some of them drawn

from statistics, some of them drawn from organization theory, a lot of different tools. So someone who wanted to go into that field, what advice would you give them? I would say, Um, that there is no clear path to

where I am right now. Um that it's it's it's not clear to me, Um, where you would go because the work I'm doing is so weird and interdisciplinary it doesn't fit into any of the existing university niches, Which is kind of funny because most university niches have become more and more specific and more and more narrowly focused, And you're going in the opposite direction, pulling from three distinct plus the whole quantitative side of it, three distinct

areas of practice with a heavy math overlay. Yeah. I don't claim to be much more general than many of the specialists my specialist colleagues. I think I'm just more specialized in a weird way. I mean, I work is very specialized and focused. It just draws on different components of different disciplines in a very focused way. Um, so that's intriguing. I think people who do in new discipline work,

you know, they're not Leonardo da Vinci. I mean, there aren't any Leonardo DaVinci as far as I can tell right now in the university world. Uh. Um, what we what we do is we we we we and we need to carve out a very specialized research programs that deliver have tangible deliverables. Uh that we were really on a tight accountability leash in this forecasting tournament. It was you know, we had we were submitting forecast nine am

Eastern time every every day. It was. It was a very rigorous process and we needed to have very focused group, tangible deliverables and a focused process that that seems like that's of great value to both business and government. Final question, what is it that you know about forecasting today that you wish you knew when you started down this road thirty years ago? Well? What I wish I knew? What I what? So the early work was mostly about cursing

the darkness. It was about cognitive bias and how we're prisoners of our preconceptions, how we have a heart that we're too quick to make up our minds were too slow to change them. It was a rather dark purtrait of human nature. UM. And there's some reason for being pessimistic, given the way we think about UM politics and history and economics for much of the time UM. The later

work has been more about lighting candles. It has a more upbeat flavor that there are specific things you can do to become more open minded, at least about relatively near term futures. And if you can become more open minded about relatively near term futures, maybe you can become a little more open minded about medium and longer term futures. Maybe you can be better able to see how alternative

perspectives might have some merit UM. And I think that when you feel you're in competition with the other side, and the other side might be getting to the truth faster than you, that has a very salutary effect. I think it will tend to make us a bit more open minded. Thank you so much for being so generous

with your time, Professor Tetlock. Uh. We've been speaking with Professor Philip Tetlock of the University of Pennsylvania UM both Wharton and other schools, author of super Forecasting, The art and science of prediction, as well as expert political judgment, How Good is It? How Can We Know? And a number of other books. If you've enjoyed this conversation, be sure and look up an Inch or Down an Inch on Apple iTunes and you'll see all of our other

eighty three or so um previous conversations. I want to thank Mike bat Nick for doing the deep dive and helping me on the research with this. I'm Barry Ridhltz. You've been listening to Masters and Business on Bloomberg radioh