Silencing the ‘Noise’ Behind Bad Corporate Decisionmaking

the school of behavioral economics. With his collaborator Amos Verski, he also wrote what I'm sure is the best selling psychology book of all time, Thinking Fast and Slow, along with Olivier Siboni and cass Sunstein. He has a new book just coming out in paperback, Noise The Flaw in Human Judgment. Olivier is a professor of Strategy and Business Policy at h GC Paris, and I'm pleased to say he joined us for the interview. But I started by asking Kannerman to tell us what this great flaw in

human judgment was that he caused noise? Define uh, what noises in relation to its cousin more familiar cousin, which is biased? Uh. And we're talk in general about errors of judgment, and the context when we talk about errors of judgment is to compare judgment to measurement. That's where the whole notions of accuracy and accuracy of measurement and error. The regulous treatment of errors stems so com from the

comparison of judgment to measurement. Now, in measurement, when you're measuring the same object multiple times with the very fine rulers, we're not going to get the same result every time. That is, if the ruler is fine enough, there's going to be variability. That variability, that's the variability that we call noise. So the errors some areas are positive of the errors are negative, and the variability of the air a noise. The average error is a bias, so that

you can have positive bias or negative bias. And it turns out that in the discussion of errors of judgment we have focused on systematic errors on biases, and very little attention has been paid in the discussion of error and two noise to variability. But in the theory of measurements, bias and noise actually have equivalent weight, and there is reason to believe that more inaccuracy and judgment is often due to noise than to bias. And this is you know where the book came to be. And I guess

it's worth saying. We're used to thinking of the bias that you'll be systematically leaning in one direction or another in the directions that we in the decisions that we take. But I guess the point of the noise is that it's not predictable. And I guess crucially we fail to understand not just that there is noise, but the stint of the noise. We ask experts something and we expect there to be only a relatively small variation in their decisions,

and actually there's a huge variation. So I guess it's it's worth talking through one of those examples just to

give a sense of what you're talking about. I can describe the example from what the study began, and this It began about eight years ago when I was doing some consulting in an insurance company and I conducted a fairly between experiment that today we would call a noise audit, where cases were constructed which were very common representative of the work of underwriters in that company, and then the same cases were presented to several dozen underwriters about fifty

years I recall, and we looked at the variability they looked at. They put a dollar value on those cases. Now, notice those cases worth were fictions. They were constructed as such, but they were very typical. And the idea was that if underwriters vary in their judgments of those hypothetical cases, they would also vary in their judgments of real case. Now, I ask executives in the company a question that I think anybody was listening to this would also ask themselves.

If you look at two underwriters and you should pick that random and our large the difference do you expect to find between them in percentages? That is, you take the two underwriters, you compute their average of their judgments the difference of their judgrens. To divide the difference by the average, what percentage do you expect most people or many people? There is a really common answer to that question. People expect about them pass and this was also true

of the executives in that company. We don't expect judgments to be perfectly, but we expect them not to disagree wildly. Now, the real number that we observed in the experiment was about fifty five zero five times larger than expected, and that's really the origin of the book. So it looked worth studying, not only because there was a lot of noise, but because the noise came as complete news to the organization.

They were unaware that they had a noise problem. So we started with slogan, which is and it turned out that there's a lot of noise everywhere, that wherever there is judgment, there is noise, and there is more of it than you think, and that is really the motivation for the book. Olivia joined me very soon and they started working in the book together. Cast joined us later. Um, that's the story in a year ago the book. It

is interesting. You know, obviously you're from different fields and Olivia, I'm interested in you are drawn to this because obviously it has very clear relevance for business strategy and the way companies think about sort of what it is they're doing. Well. As Danny has just described it, noise is unwanted variability in judgments. And this only becomes a problem when you're an organization. Noise is a disease of organizations. If you are an individual, we will never know how noisy you are.

You are noisy, by the way, and you are you are subject to the same sources of noise that we are all subject to it organizations. But where we expect consistency is when people in an organization are making judgments on behalf of the organization, as in the example of the underwriters that Danny was just talking about, and when

we expect those judgments to be reasonably consistent. If you look at another example we've looked at, which is the judicial system, we expect that the decisions that judges renders should not be too dependent on the identity of the judge. Of course, again we expect some variability, but we expect general consistency. And the challenge for organizations of any kind is to actually achieve something approaching consistency, because first they

need to realize how much inconsistency they have. They need to realize how much noise there is, and as then he pointed out, they're not aware of that. It comes as a complete surprise when they realize that. So it's a huge organizational problem for private enterprises, but also for administrations, for government, for non government organizations, for any organization of any kind that has many people making judgments and that

expects consistency. There's so many different strands of this. I think there's one which is a straightforward sort of natural justice perspective. But some of the examples in the book, there's some of the ones that perhaps were most familiar with is the variation in sentencing for the same case depending on whether the judges football team one or lost

at the weekend or whatever. It maybe. But it also raises a question about what is the nature of expertise If we like to think that an experts are experts in part because they understand the body of knowledge and have a shared understanding of that the way the world works in that particular expertise, And what a lot of these examples suggests is that they're all experts in their own way, and they're all coming up with completely different conclusions.

Does want come away from this thinking that experts are not necessarily helpful for organizations. I think you've come away thinking that there are really two different sorts of experts, and that we should be clearer in our articulation of that distinction. There are experts whose track records can actually be evaluated, whose expertise can be quantified measured against a

gold stand earth. So if you're a forecaster and you make short term economic forecasts, and each quarter we can check how oft you were and it turns out that your forecast historically been very good, we can say you're a true expert as forecasting. Now, if you're making thirty year forecasts, how much of an expert you are does not depend on how good your forecasts are. It depends on how much respect we accord you as a forecaster.

And those are the experts that we call respect experts because they're experts, not because they have demonstrable experts ese, but because they have convinced others of their expertise, because we have respect for their expertise. That is not a criticism of those experts, by the way, because in many fields, all you can be is a respect expert. Even the underwriters that Danny was talking about, we'll never know if they've actually set the right premium for an insurance policy.

So we have respect for them because they are convincing, because they can are securely their reasoning in a compelling manner, because they have experience, because they have gained confidence in the way they do their job. But they are not the same sort of experts as the experts whose expertise can be demonstrated. And what we argue in Noise is that it's important to know what sort of experts you're dealing with. When you're dealing with experts, Danny kind of.

And you talk about good decision hygiene as being the sort of equivalent of washing your hands so that you can have the limited infection from noise. What what does that look like for a for a policymaker or an organization. Well, well, whole notion of the hygiene is in contrast to common efforts of the biases trying to reduce various biases in the thinking of the judgment of organizations and individuals. Uh, the bias thing is very much like medication or vaccination.

It's specific to a particular disease. Hygiene, like washing your hands, is non specific. But as you don't know what germs you're killing, if you're lucky, you'll never know. And uh, that's that's the nature of hygiene. And when you think about noise, the only way that we could think of improving judgment of reducing noise is by taking steps which are generally steps to improve the quality of judgment, but are not oriented to particular biases or two combat particular biases.

What would be a good example of good hygiene where you could otherwise have a very noisy and unfair decisions. A standard example, and actually an example that was very influential on our thinking, is how to conduct hiring interviews. And there has been a lot of research on hiring

interviews and they fall into two broad families. Unstructured interviews, that's the common procedure where you talk to the candidate, you try to form a general impression, You have a conversation with the candidate, there is some human contact, and at the end of the process you you make a decision or you form an impression of that candid. A structured interview is very different. In a structured interview, you have a list of topics that you want to think about.

For example, you want to assess various attributes of the candidate, how original, how reliable, many attributes that may be relevant to a particular job. And in a structured interview, you think about each of these areas in turn and you conduct an interview. That is, you ask questions that pertain to that particular area, actually write down a grade or ranking or rating for that before switching to the next topic.

So that's a structured interview. Now it turns out that neither kind of interview is very good because because basically performance in on jobs is very difficult to predict and it doesn't depend only on the characteristics of the individual. But structured interview are distinctly superior to unstructured interview. And so structuring is we think a good idea, and when you're making a decision and you're considering various options, you might want to consider the options as if there were

candidates and assess the various attributes of an option. And the important feature here that you delay the global intuition. You delay the formation of the global impression. Intuition. One of the problems of intuitive thinking that it comes very fast. They form first impressions, and in unstructured interviews, typically an impression is formed very quickly and most of the rest of the conversation is to justify the national impression. In

a structured interview, that's not the case. You deal with topics one at the time and you try to delay the global view of the candidate until all the information is So that's an example of decision. But I mean, one of the points about intuition is it's not very controllable. So I'm just wondering. I mean, you know, you and Olivier and and and behavioral economists may be very aware of all the biases and all the noise that you've

just talked about. But when you're sitting interviewing a candidate or a potential colleague at university or whatever, how do you actually stop yourself from having a first impression of someone, Because by definition of first impression comes unbidden. Oh, you

will undoubtedly form impresference, there is no question. But but if you have a set of questions that you want to ask about the person's reliability or about the extent of their experience on similar on a similar, unsimilar task, those specific questions are going to fail your mind, and they're going to push the intuition aside to some extent, and you will have an opportunity that you normally do not have of disconfirming your initial impression, of finding things

out that actually do not fit the initial impression. In general, an unstructured interview, impressions are self reinforces. You justify your initial impression, and that is a source of noise, And by structuring the process you reduce that source of noise. An additional thing you can do to limit the problem that you're pointing out Stephanie is to have different people or different sources of information evaluate the different dimensions that

you are looking at. So, if you're evaluating candidates for you know, intelligence, technical skills, and fit with the culture of the company, let's assume these are the three dimensions

of your job description. In an unstructured interview, you would form an overall picture of the person and you would then raid them on the three dimensions, but they would be strongly correlated with each other because there would be a positive or a negative halo around the person, and you would say they are great and everything, or they're

bad on everything. Now suppose that we say you, Stephanie, are going to conduct the interview about the technical skills, or maybe in fact, we're going to have a technical test to evaluate the technical skills. Someone else is going to evaluate the fit with the company, and someone else is going to evaluate how smart the person is, or again, perhaps we're going to have the test of how smart

the person is. Now you've got three independent data points that do not influence each other, and you have a much more structured process to make your decision. You would have anounced this before, but of course, a lot of people entering the job market now find that they're at least the first couple of rounds, depending on how popular the job is is, and they're talking to a computer or they have there is a an AI element to

their application process. We may not like it, we may think it's not true to our great sense of intuition about people. But from a fairness perspective and from a decision hygiene perspective, is that a better way to go? We need to be careful here, because there is an answer in principle, and there is an answer in practice. In principle, any form of structured decision making that reduces noise would in fact enhance the quality of the decisions.

So if you have an algorithm making decisions, there is going to be less noise there. But of course the question is how good is the algorithm? How good are those AI systems that people sit in front of, And I'm sure there are good ones, but from the ones that I've seen personally in my admittedly limited experience, there isn't much evidence, and there isn't very good quality evidence that what these software packages are testing for is actually

what you're looking for. It's actually quite hard for most companies to define what it is that they're looking for, and there is no evidence that I've seen that there's any correlation between when those software packages look for and job success is actually highly correlated. So in practice I'm

quite skeptical about what I see in the market. In theory, I have to agree that it makes some sense, but my worry is that companies are using this mostly to save time and money, not to actually improve the quality of their decision. Here, I would hope that what is true in theory can be made true in practice. And one characteristics of algorithms and is that they're improvable. They're much more improvable than people are, and and they can be corrected by by data on quality. They can be

made to predict more accurately. So this this is really an issue of the quality of constructing algorithms. And there are many algorithms that are of poor quality out there on the market, and there is a widespread suspicion of algorithms which makes us prone to reject them. But by

and large, I think this is the future. In the future, there will be more and more of those algorithms, and their quality will be getting better and better every year because there will be data there will be feedback, and the feedback can be incorporated into an algorithm much more

efficiently than it can be in the human judgment. So uh, here I join Olivia's skepticism about most of the algorithms that exist, but I really want to register and mode of optimism about the future of that kind of operation. There's one trend which is about eliminating the human element to some extent, or at least having it in a more regular form in an algorithm, a more consistent form structured.

Of course, the other big trend in business strategy and conversations about companies is the move is encouragement of diversity and to encourage businesses in a sense to have a wider variety of humans doing the judgment. And I wonder whether that even goes against some of the things that

you're talking about. You know, one of the ways that companies might have previously eliminated noise, not necessarily error, but noise, would have been having lots of identical people or making the decisions all of these white men sitting in their boards. If you now have a greatly much more diversity, you might be more true to the range of human experience,

but you'll be getting a lot more noise. Well, that is certainly true, but in in principle, we we want to distinguish between the process of generating a judgment and the final judgment. In the process of generating a judgment, diversity is very welcome. That is, you want multiple points of view, you want people ptise to enter into the participate in the conversation. But when a final judgment is made,

we want a process that reduces noises. So diversity is very useful, and you know it's it's you can think of that in terms of, say, witnesses to a crime. So you're better off if the witnesses are in different places and see the event from different perspectives. And you're certainly better off if if the witnesses don't talk to

each other and they are independent of each other. And so thinking along those lines gives you an idea that you do want diversity, but you want also the kind of independence and the kind of goal directiveness that reduces noise in the final journey. Diversity in the outcome of these decisions, in the judgment that you produce in the end. It's good for some things, but for most it's not. When you when you go to the doctor and the

doctor tells you, oh, you have is disease. And then you go to another doctor and he tells you you have that disease. You don't say, oh, that's wonderful, it's diversity. You say one of these two doctors is wrong, maybe both. So whenever we think that there is a correct answer, diversity in the outcome is not good. Maybe the way to get to the correct outcome is to harness the diversity of the perspectives or through multiple witnesses, and that is one of the remedies that you can have to

reduce noise. But as an organization, what you're aiming for is not every person having their own opinion. It's any person having the best possible judgment. And we've talked about business, we've talked about justice or you know, sentencing and decisions within the criminal justice system, and an area that comes up a little bit in your book, but obviously it's kind of front and center of people's minds at the moment when we think of the decisions being taken around

the war in in Ukraine. Is that in a critical moments of foreign policy or military strategy decisions, you know, you can't necessarily enlist a lot of people and listen to their structured answers on a set of questions in reaching your judgment about how to respond to Russia or how to how to respond to one of the sort of very pressing situations that can arise in foreign policy.

So I just wonder whether you whether you'd reflected on that, you know, if you're Anthony Blink in the sector of state, or if you're President Biden, or or that matter, at a Russian general how what does decision hygiene look like in those kind of situations where there's inevitably going to be a limited number of people that you can call on and imperfect information hygiene is something that an individual can follow. That is, there are better and less good

ways of individual judgments. You want to cover all the bases you are to think, You want to the extent possible to think of all possible consequences, you know, as salient example in the Ukraine War, is it looks unlikely that that people who started that war knew that Finland and Sweden would want to join NATO, because you know,

after all, this was supposed to keep NATO away. So when I have the feeling that not all the bases were covered in making those critical, so there are all we can hope for is that we have people with a lot of experience, because it turns out that there is genuine intuitive experience that can develop over time with institution within certain kinds of decisions and choices. And we also want people who, even under sometime pressure, UH, can

follow the basic dictates of decision hype. I've noticed a professor kind of and that a lot of interviews with you are quite long, a long than the sort of the norm for whatever program it is, And I suspect it's because it's so it's so it's always so fascinating to listen to you, and you always want to have another question. But we're going to run out of time. So thank you very much for coming on Stephonomics, and thanks thank you to Olivier Simbony, thank you, thank you

very much. That's it for this episode of Stephonomics. We'll be back next week. In the meantime, do please rate the show if you like it, and check out the Bloomberg Terminal and News website for more economic news and views on the global economy. You can also follow our economics on Twitter. This episode was produced by Magnus Henrickson and Summer Said, with special thanks to Professor Daniel Kannerman and Olivier Sibon. Mike Sasso is executive producer of Stephonomics

and the head of Bloomberg Podcast is Francesca Levi. The