Hi, I'm Aaron Welsh and this is This Podcast will Kill You. You're tuning in to the latest episode of the tp w K Y Book Club, where I chat with authors of popular science and medicine books about their latest work. Since starting this series a few years ago, I've gotten to cover some amazing books and I appreciate so many of you reaching out with your suggestions for
books to feature. Keep those recommendations coming, please, and if you'd like to take a look at the full list of books that we've covered in this series, as well as get a sneak peek at ones that are coming up in future episodes, head on over to our bookshop dot org affiliate page, which you can find on our
website This Podcast will Kill You dot com. Under the extras tab on the bookshop page, you'll find several podcast related lists, including one for this book Club and the TPWKY Kids book Club, which if you're not following us on social media you absolutely should be because Aaron Epdyke
has been putting together videos reviewing children's books. It is such a great resource for sciencey kids books for all ages and if you want to share your thoughts on these episodes, make topic suggestions submit a first hand account. You can get in touch with us using the contact us form on our website. Two last things before moving on to the Book of the Week, and that is to please rate, review and subscribe. It really does help us out. And second, you can now find full video
versions of most of our newest episodes on YouTube. Make sure you're subscribed to the exactly Right Media YouTube channel so you never miss a new episode drop. Belief is a powerful force. It shapes every facet of our lives and transforms perception into reality. What we believe to be true is not always what is actually true, something I'm sure we can all relate to. Maybe you've debated with a friend over the answer to a trivia question, like you both know the right answer, but your answers are
somehow different. Or maybe you've had a heated exchange with a relative who firmly believes that the moon landing was faked. How do we decide what we believe? How can we know that what we believe is the truth, and how can we convince others of that? These are precisely the questions that Adam Kucharski, who is professor at the London
School of hygiene and tropical medicine asks. In his latest book, Proof, The Art and Science of Certainty, Kucharski, who is a mathematician that works on infectious disease outbreaks, explores how we are inundated with information and increasingly misinformation, that we have to evaluate to determine whether or not we should incorporate it into our decision making. This extends beyond personal decisions which root is best to take to work, what to
make for dinner. Our world is built upon structures of proof with varying degrees of support. That car that you drive to work is manufactured under rigorous safety testing, meaning there are established guidelines for what is considered safe and how to test that same thing. With the food we eat, the medicines we take, the buildings we spend time in. We don't question so many of our beliefs. To do so would leave you frozen, uncertain of which direction to
move in, what to trust. You'd have no time to actually live your life. But when we do scrutinize our certainty, we might find a gulf between our beliefs and someone else's, and those beliefs and the objective truth. Where does that incongruity originate. Why are we skeptical about some things and not others? What does it take to make up our mind and what does it take to change it? That
answer might not be the same for everyone. An enlightening blend of philosophical musings, political commentary, statistical exploration, and personal reflection. Proof is a fascinating read, particularly as this unceasing flood of information, both good and bad, shows no sign of stopping. Let's take a quick break and then get into things. Professor Kochowski, thank you so much for joining me today.
Thanks for having me.
I am thrilled to talk with you about your latest book, Proof, The Art and Science of Certainty. And before we dig into the various forms of proof and how we determine a threshold for proof or what different types of proof exists for certain situations, I want to start at the very beginning. What is proof? Is there a standard definition?
Yes?
I think that's a great question, and that I think.
My background's in math, so I think a lot of MICA training was around this idea that you can have this definitive knowledge that something is true, and I think it's something that people grappled with across fields I mean, more of the stories that really struck me was Abraham Lincoln, when he was training to be a lawyer, came across this word demonstrate and yeah, this kind of beyond reasonable doubt, this certainty, and he's like, I don't really understand what
this is as a concept, and he actually went back to all of these ancient Greek mathematical texts to understand how can we take what the knowledge we have, build on that prove new theorems, use that to prove subsequent knowledge. But I think one of the things that was really the motivation for the book, and something that I think anyone who works with information and decision making and evidence happens across very often is it can become quite a
shifting concepts. I mean, even in mathematics, things that people thought were proven turned out had some hidden assumptions or human judgments that were kind of lurking there and caused
a lot of that to collapse. So I think it's it's a kind of fascinating concept because it's something that's so important in life, not just having knowledge that we gradually accrue, but from many of the things we care about, whether it's dealing with them, emergency, whether it's a legal case, whether it's even just a kind of minor business decision in our day, we have to work out where we set the bar and how we evaluate what we've got. And I think for me that was really the launching
off point to explore this. You know, how do we converge on certainty and what happens when it goes wrong?
Thinking about the difference between proof and certainty and truth, like, what is the relationship between those concepts?
Yeah, I think that's a great question, and without going down the kind of philosophical rabbit or it could have been a book.
On you what is reality?
Yeah, But I think the way that I approached it is just to look at how people thought about this in different fields. And again even going back to Lincoln and much earlier, there was this this appeal of this certainty, this idea that there could be this universe truth, and it's why a lot of fields ended up borrowing for mathematics. You see it in the US Declaration of Independence. Yeah,
we hold these truths to be self evident. The visional draft was we hold these truths to be sacred and undeniable. But Benjamin Franklin didn't like that, because it sounded like they were kind of appealing to some divine authority and self evidence is just borrowed directly for maths, it's just a given truth. And unfortunately it turned out a lot
of these things about equality weren't self evident. But I think that that story of how you think about these things, and even when we see in the legal world, a lot of it was originally derived from concepts around matter,
around probability. If you talk about, you know, some of these thresholds preponderance of evidence, you're saying it's more likely than not, and you're kind of borrowing a lot of these kind of probability based ideas, and even in the world kind of more experimental design as that kind of developed, a lot of it was about I mean, actually some of these early studies were almost trying to discount some
of the influences of religion. You're wanting to understand cause of effect in the world, rather than just appealing to some other influence.
And then it for a lot of people it became a.
Discussion of how do you take the evidence you have and how do you link that to a conclusion that you want to make, and where do you set the bar for that do you try and get ever closer to certainty? And there's actually a lot of statistical tension about one hundred years ago, no statistical debates. Kind of
sounds a bit boring, but it was actually this real. Yeah, people just almost I wouldn't talk to each other because there was this tension between do you just try and get ever closer to the truth or do you have
a framework that allows you to make decisions? And I think a lot of times in life we don't get to do the academic I'm just going to sit on the fence yet I just I don't know, and I'm just not going to do anything with life or actions that often we have to decide we do something or we don't do something, or you know, we say someone's guilty or we let them go three or there's these
decisions we have to make. And so that process of interacting with evidence is much more pressure and I think that that was one of the real big tensions that never fully got resolved.
Actually, even how.
We teach statistics at school, we kind of smushed together these two very different philosophies, one of this ever higher bar for evidence and one where we're sort of outlining a framework to make a decision based on the knowledge we have.
When it comes to public health and medicine, there's a lot more pressing, you know, need to make decisions, and yet this decision is often dragged out for long periods of time, and sometimes that is at the urging of, you know, someone who has incentive to drag out a decision.
So one of the examples that you talk about is Austin Bradford Hill, who is talking about this relationship between cigarettes and lung cancer and saying, oh, we have the we have some evidence, and there's still a lot of skepticism, but we have enough to make a decision. We cannot
use uncertainty as an excuse for inaction. Do you feel like that, like we've ever truly learned that as a society, or has it been you know, players like the tobacco industry saying oh, no, this uncertainty, you know, we need to push for more and more and more evidence.
Yeah.
I think that's so, that's a really good question.
I think that's a really good example of almost kind of weaponized certainty, that you can always set the bar higher, and any aspects of life, you can set the bar high and higher and higher to the point where you just won't do anything. And in action, of course, is in itself a decision. And I think Bradford Hill's work, you know, he was extremely thoughtful in how we approached this because something like smoking, you can't really design it like a try. You can't get people to randomly take
up smoking and see if they get cancer. There's obviously ethical reasons about there's also just timeline reasons. You know, if you look at the timescale of the intervention versus what happened, you might have to wait decades to have that clear signal. And so he did a lot of poning work with others, linking together the various sort of
non random data sets you had available. Because one of the criticisms, of course, as any days, is yes, smoke is more like to get cancer, but maybe there's a genetic reason that makes them all like to smoke and
get this. And he outlined a lot of the ways we can think about cause and effect, and I think that's a very useful set of concept and some of it the obvious ones of the cause that needs to come before the effect, or that you know, if you have the strength of association more of cigarettes makes you more likely to get cancer, if you see that across multiple countries, or if you can start to think about, you know, the biological plausibility. We see carstinogens in other
kind of situations as well. None of those things on their own is conclusive. You can start to build this evidence space. And he made this really good point that any knowledge we have, even if it's very confident knowledge, is always subject to further refinement. But we still have that knowledge at that point in time, and we can seek further information. There's been lots more studies of smoking since they're early ones, but also that's information that we
have to do something with. And I think we often particularly in the situations with emerging threats or kind of early concerns about things, whether it's a health intervention we think might be harmful. I mean, what of the examples are given the book is the work at the FDA around flidamide, which was this for sickness in pregnancy, and there was actually a lot of concerns about safety for babies, and the FDA blocked it as a result. But on the other hand, you get things where there might be
a lot of value. For example, in reducing smoking for health outcomes, and even if there's an uncertainty, And Bradford Hill made this nice point of actually the standard you should apply for taking action kind of depends a bit on the situation you're dealing with, if it's a fairly cheap action to take, if it's not too disruptive for people. But actually, in his argument he said, smoking is something people really enjoy, so we need a kind of.
Higher barrier if you're get it.
And I think it's a reasonable point if you're going to tell a lot of people to change how they live their lives, that the evidence you need is perhaps different for something where you can take some action and you can unwind that. So it is those kind of trade offs that you have available that obviously need to play in as well.
Let's take a short break and when we get back, there's still so much to discuss. Welcome back everyone. I've been chatting with doctor Adam Kucharski about his book Proof, The Art and Science of Certainty. Let's get back into things right. The thresholds for certainty is it can be different depending on the situation, and then there's also these personal thresholds for certainty or evidence, you know, how much
information do we need? And one of the things that you discuss in your book as well is sort of what happens when evidence flies in the face of our personal beliefs, and how sometimes even despite a mountain of evidence, we can just still feel like that's not possible. We can't reject it. It's not an intuitive truth, you know, what happens, Like what is show us about sort of the personal nature of proof and certainty.
Yeah, I think that's one of the things that really kind of struck me in researching that. I mean, even in some of these kind of mathematical puzzles examples, it's things that you know, I'd come across as a kid and convince myself, Oh, that's just that's the answer to the puzzle.
And it was only years later when I was explaining it.
To someone else or someone else had asked me about it, and I sort of went through the thing that convinced me, and it just didn't convince them at all. And I think that's really interesting. I think we focus a lot on, you know, how science works, how methods work, what convinces us, and you see this in even a lot of studies around political beliefs that people will often try and convince others with arguments that convince them, and then you get
this gap and it's almost like that just fails. And I think that's a really interesting step to explore, well why does that film. One of the things that I find even just kind of in some of the modern tools we have in the modern there we are quite striking is where we have this desire to explain things. So, yeah, few years ago to talk to a bunch of people working on AI, and there's a lot of concern about I think things that self driving cars that we don't
understand why they make mistakes. We need that explainability. We can't have things we don't trust. And actually, in medicine, we have all sorts of things that we know work. We now often they work, we don't fully understand the physics and biology something like anesthesia, for example. We can control the effect it's going to have, but actually all the underlying biology and kind physics mechanisms is still more
work to be done. Things like defibrillation. You know, if you give a heart a shock, you can kind of reset it against Some of that's understood, but there's still those kind of gaps in knowledge, but we know that these are useful tools. And even yet, if you run a clinical trial, you can assess how effective a treatment is, but that on its own will just tell you the effect. It won't tell you necessarily all the mechanisms that are
going on to explain it. But there's these tools that we've got, and we've got the evidence to take action and use these things.
We're very happy with.
And there's other areas of life where actually that inability to explain something kind of really bothers us. You know, even if self driving cars were much safer other than humans, as humans we are start looking to a book, humans are not good at driving. You know, it's not a massively high bar, but I think it would still make
people uncomfortable. Even if they would say twice as safe in cities were as well very well defined, I think it still bother people if every now and again there was just an accident that we had no real idea of what was happening. And I think that's really important to bridge because I think that particularly when you get that gap in understanding, that's room for other explanations to
kind of creep in. And I think that's where we start see emergence where things like conspiracy theories, whether it's things with kind of incorrect logic, Often it is that gap between what we're seeing and the understanding of why that's happening.
I think humans have this.
Very in many ways, very powerful desire to explain what they're seeing. But in some cases where the explanation is very hard to untangle, it can lead us astray.
That's fascinating to think of that, the gap between understanding and what is happening. We don't understand how anesthesia works, or how Thailand are a menif it truly works. But we do understand how vaccines work, for instance, and yet there's so many conspiracy theories and misinformation surrounding this thing that we do know how it works. I guess what good does evidence do if we do not take it into account and are not open to it.
Yeah, And I think for me, a lot of it is just understanding at what point that breaks down. I mean, even if you if you look at some of the COVID vaccines for example, you know, or or even some of the kind of other debates around climate interventions. I think often it gets very into debating some element of the technology, and I think often it's actually just people disliked some of the control that was exerted over them
through mandates or for other things. And actually, you know, if you've got an intervention you're unhappy with, you can disagree with the intervention of saying look, yeah, for example, we know that intervention works, but I disagree with how you're implementing it, or you can disagree that the intervention actually has an effect, even one step down and just say you know, actually, I think there's there's sort of
deeper problems, or maybe the disease isn't a threat. And I think often those kind of levels get get tangled up, and I think in a lot of conversations I've had with people, often they're sort of deep down concern or the thing that they're approaching it with isn't necessarily that they've just out of nowhere decided that this isn't a threat,
or that that technology doesn't work. It's actually in some of these instances things are a bit more marginal, and you could say, you know, you can make an argument either way even if the underlying intervention is effective or is going to have this You know, you can make there's moral and this it's not just about an sort of epilogical question, as I think, kind of understanding where those drivers are and also just in our own arguments.
I think sometimes, you know, I have the conversation with people and I think I'm just arguing about the kind of the nuances of whether intervention is a good idea or not, and they're actually arguing whether it's a problem in the first place. We see it vaccines, I guess examples more polarized, but even something in climate You know, you can have a lot of people who just agree
on the nature of the effect of climate change. They agree on the different levers that we probably have available a society, but they might strongly disagree about actually how we prioritize those and all of the trade offs. And I think it's just understanding what level we're on and where the evidence might stop and where it might then just be other things that are filtering in on a personal level.
This idea of proof and certainty and truth. It seems very intuitive in a lot of ways today, but this maybe wasn't always the case, Like when did the concepts of truth and the need for these self evident truths or certainty or proof when did these come to be? And then you know in what fields or what areas where they initially applied.
Yeah, I think that's a great It's easier just to think of like the world and sort of science and evidences always was as it is. I mean even in mathematics, it's idea that we had a universal truth wasn't the same throughout history. If you go back to the ancient Egyptians, ancient Babylonians, they were much more focused on problems solving. A lot of their texts are kind of these these kind of puzzles and very much things around kind of
practical everyday problems. And even if you look at their formulas for an area of a circle, they're quite approximate, and if you're building something that needs quite a large circle, you're probably going to be okay using those, but it's not going to give you that really precise truth no matter what problem you're working on. And that's something where the ancient Greeks mathetations like Euclid came in and tried
to put things on much more solid footing. So you've got these concepts like pie that if you want the area of a circle that will just be universally true and you won't have this issue of your kind of approximation breaks down. And it was then, I think that as it sort of came into the Enlightenment here, it was very appealing for people that you could have these undeniable truths about the world. And I think that's where a lot of other fields started growing them as well.
But even in medicine, if you look at this study of cause and effect, a lot of that it was the sort of medieval Arabic world that a lot of that started to emerge. So a lot of the kind of superstition, this idea that disease or conditions just kind of come out of nowhere and it's bad people or you know, someone's a witch. All this kind of stuff that was going around in much of Europe at the time.
There's a lot of early writings inn around sort of the eleventh century saying these aren't supernatural, there's natural causes and we can study them. Yeah, we can study them, we can work out what the cause of effects were a lot of early attempts to try and think about concepts that we would now call things like having a control group or thinking about how we kind of you know, would divide and treat some people and not treat some people,
and then compend the difference. The conclusions didn't always work out. I mean, there was I think one of the earlier studies was someone who'd identified correctly the symptoms of meningitis, but then concluded that blood letting was really effective for it, which probably something in their study design had gone astray. But again, it just kind of really and it's one of the things you look back on and you think it's just it's pretty obvious that we should be doing
it that way. But even coming into the twentieth century, if you look at something like analyzing a medical treatment, a lot of the early studies did an alternation method, because if you think about it, rather than randomized patients, you could just say, well, the first patient that comes in, I'm going to treat, the second I won't, the third I will, forth they won't, and on average you should get something that any other sources of variability should balance out,
and the difference between those groups should be on average down to the treatment effect.
But Bradford Hill actually, who did a.
Lot of the pilneering work in the early clinical trial space, noticed that the groups were often imbalance because what's happening as patients were coming in and doctors were maybe subconsciously that, oh, maybe that person looks a bit ill, I'll enroll them, or maybe they don't meet the diagnosis it And actually a lot of the early randomization wasn't statistical. It was just it was to sort of keep humans from themselves
because we couldn't trust subconscious judgment. So a lot of the early randomization in medicine wasn't about the statistical properties of the trial design. It was just about making sure human didn't muck things up basically with their internal biases.
Well, I mean, we'll find a way, I'm sure somehow in some way. It's interesting to think about this idea of like self evident truths thinking back to you okay, yes, or superstition and this person is a witch based on these signs or whatever was that also viewed as proof.
The story of those trials by old deal is a fascinating one as well, because they were used for a long time. You could have trial by ord deal like by water by fi whatever, and then you could also chose trial by duels. So basically the big criminals always pick that, and people start to notice like, oh, you know, if God is deciding which one's innocent, God tends to pick the bigger one like pretty much every time, which.
Is I think there was that that came.
But actually one of the reasons they stopped using them is a lot of the religious scholars became concerned that they were basically trying to by running those trials, you're essentially trying to get God to do your work for you, and that felt for them a bit awkward because you're sort of on demand saying, hey, can you come and make a decision for us, which they it sort of got quite uncomfortable with. But even those early systems, I mean early juries in England were kind of fascinating because
they they weren't the structure that we had today. They kind of did their own investigation. So often someone was accused and then they went off and accused someone else and kind of did their own thing, and it was it was only over time that system kind of evolved of having that way and converging something and I think that's we talk a lot about the problem of black boxes, but to some extent, juries and talking to legal scholars was kind of interesting with this that it's not so
much about getting to the truth. It's having a system where you can reach a decision and you've got kind of that finality or send me from finality to that decision, and having a system that works, rather than you know, your one hundred percent convinced of that. And I think we see that kind of across different fields, of that emergence of truth. And you said, what's kind of obvious and what self evident? I mean, one of the other things that I found kind of interesting was how many
mathematicians were deeply influenced by religion. So Newton, for example, Isaac Newton, driving all these equations and theories about planets and planetary emotion, he saw that it was God keeping the universe in balance, and he was essentially just observing divine influence. So for him, although he was doing a lot of this scientific work, he saw that there was this external influence keeping it all in place along the way. So even quite far through history, you had these kind
of other baseline explanations going on. I think even in the modern era. I think the way sometimes we tell the story of science I think is sometimes almost a bit disingenuous. If you read a scientific paper, it's kind of yet there's this problem, and I decided to run this experiment and I got these results. But I think there's also just that element of what was the hunch that made you think that that might be an interesting
thing to investigate? What was that spark of inspiration? I think, even in this era of AI, it's a really interesting question, because AI can kind of process and mimic human decisions as we write them down. But I think there's often that kind of spark or that idea that would lead you to do something that just people wouldn't have tried before. And that's much much harder to articulate. So it's not necessarily that kind of obviousness that we might have had
in another era. But I think there still are those things which are quite hard to explain in where that evidence might have initially sparked from.
One of the things that you mentioned was the use of proof and evidence in the legal system, and I feel like this was a really fascinating discussion in your book as well, where this is employed as like, you know, proof beyond a reasonable doubt or innocent until proven guilty. What does this show us about like the variable level of evidence needed to make a decision, and I guess like the different forms that proof can take in this setting.
It's a really interesting question about how different societies have even said that that balance me Essentially, in a legal case, there's two main errors you can make that someone can be guilty and you can let them go three or they can be in you can convict them. And William Blackstone, who is a legal scholar, in the seventeen sixties came up with wats known as Blackstone's ratio. He said it's better for ten guilty people to go free than one
innocent to be convicted. And Benjamin Franklin actually and even accortiately said it's better for one hundred guilty people to go free of one isn't to be convicted.
Seeing that as a kind of balance.
Other cultures, particularly some communist regimes in twentieth century, set it the other way. It's like it was better for ten innocents to be in prison than warner guilty to go free, because there's this kind of trade off and where they're seeing it as the worst error. And actually some analysis looking at US legal cases, obviously they don't try and target these error rates, but you can sort of infer how people are valuing this. A lot of them seem to land between that kind of Blackstone and
Franklin ratio of error. But then, of course, yes, the different evidence and how it makes it its way into the court room, particularly some of the examples historically of kind of things like early probability. And again, one of the challenges here is what What's god I talked to called the weak evidence problem, and I think a lot of how we navigate life is around probabilities that are quite likely. You know, a lot of probability theory was
originally developed around like dice games and things. You know, you can study and you can quantify. But in legal cases, we often have this weak evidence problem where someone ends up in some extremely bad looking situation from a guilt point of view, and you're like, well, it's extremely unlikely this is just a coincidence. But then if you think about it, like, well, this person might just be a normal, everyday person. That what, it's extremely unlikely too that they're guilty.
So you have these two extremely unlikely events, and a lot of statistics just isn't equipped to handle that. And so there's this notion it's called the prosecutor's fallacy where people say, well, this is the probability that that will all be a coincidence, and therefore that's the probability they're innocent, But of course you've got to weigh it against the fact that it's extremely unlikely they're guilty as well. And
we see this even in other areas. So the work we do dealing with like emerging threats and you know, pre COVID, there were some studies and actually we did a TV show where you sort of say, oh, it's you know, a pandemic could just be round the corner. Or there's another study that the World Bank I think put it at one percent, and you're like, well, what is that?
That's is that right? We was that a good prediction? Was that bad? And it's these these very unlikely events.
I think in legal cases again, for that weak evidence problem, it's less about do we definitively work out with high probability, which is true, and it's more just we have to converge on the best explanation for what we've seen given those two possibilities, and in reality we may never have certainty about where we are. And I think it's something that kind of struck me both think about that, and then also thinking about a lot of people who have
to plan for emergencies and very unlikely events. Thinking a lot of the way we traditionally think about probability can very quickly lead us astray because I think we're so used to having this idea. Well, I can be ninety nine percent sure that this happened, But actually it's much more about that balancing app that we have to perform.
Let's take a quick break here, We'll be back before you know it. Welcome back, everyone, I'm here chatting with doctor Adam Kucharski about his book Proof. Let's get into some more questions thinking about this in the context of COVID, when you know, things were evolving very rapidly, the situation was evolving rapidly, and the general public and you know,
of course government officials wanted answers and wanted decisions. You know, what is the best thing to do, wear masks, not wear masks, sanitized groceries, all these things that were just constant questions and neat and people wanting hard answers like just yes, period. And of as someone who was on the informational front lines of the COVID pandemic, what was your relationship with uncertainty, Like at that time, did you struggle with feeling like we don't have enough information yet?
You know, how did that feel? I guess in your position?
Yeah, I think I mean those those kinds of situations were normously chanted, both in terms of evidence generation and communication and then obviously the political decision making that comes off the.
Back of it.
I think in many of those situations, I found it useful to kind of convert in some cases uncertainty around the exact estimate to just just to kind of poorly what situation we're in. So for example, when I think it was the delta variant emerge and we did a lot of the work identifying the early advantage it had, and it really wasn't.
Was it for thirty percent? Was it forty percent? Was it sixty percent?
But essentially all of those were a big problem and it's kind of arguing like is this you know, is this a disaster or just a catastrophe or just very very bad And it's like right, right, Yeah, from a policy, you don't need to kind of necessarily commune out as that. You just say, like we're very confident that it's going to take off a couple of things I think that jumped out for me. I think one was the need
to triangulate across data sources. I think sometimes people have this idea of science that you go out and you run a study, and that study gives you the answer. It doesn't give you the answer. And there were quite a few of the early skepticism were saying, well, actually,
this study wasn't definitive, and this study wasn't definitive. But once you start to look at all of those, you know, you start to look at the evacuations flights, you start to look at the testing data and the contact tracing and the big testing of you know, some of the cruise ships. You start to look at the clinical data. All of those signals start to drag you in the same way. And again, each bits of those evidence on their own might have problems, but you can start to
bring together and draw that into a conclusion. I think we saw that across the pandemic that if you view it very much as like I'm going to get the perfect study it's going to give me the answer, you'll struggle. But often you can actually find a lot of complementary data sources that all for variants or a lot of that early severity were all pointing in the same direction.
I think it's harder, obviously when they're pointing in different directions, as we saw with some of the interventions, where it was less clear because different countries, different economies, certain things did affect behavior and other things in different ways.
But I think the other challenge.
That kind of jumped out, and I think a lot of the health issues we deal with in the US, UK and the modern era are non contagious, so they're very much kind of individual you know, things like cancer, things like heart disease. It's just very much individual focus. So you have someone who's ill, do you treat them, do not treat them. If you don't treat them, that's someone who's one person who's going.
To get worse.
But contagious health threats have this dependence where you know a problem can get worse and that problem can then accelerate in very different ways. I think that was something that was quite a challenge to communicate, cause I think a lot of people had this notion of you've got normal life and then you could do something else that's not normal life. And I would obviously just prefer it to be normal. But I think as we saw globally,
you didn't get that status quote. I mean, that was that was gone, and no country had They had varying levels of normality, but no country had like just you know, pretend absolutely nothing happened. You either had in varying degrees depending on the structured society and advantages they had in terms of demography and healthcare and other stuff, big changes in behavior or borders, whatever, or you saw a.
Huge amount of death.
And I think that's something that can be from an evidence point of view, much more challenging, because I think just in life, we're much more used to those kind of linear problems where you know, like with cancer or something, these are event tragic events that happen sort of distributed across the population, rather than something that the worse it gets, the worse that worseness accelerates.
You mentioned how we have these different data sources, these different you know, studies that are all leading us in a certain direction, and we have by this point in time developed ways to measure both the quantity and quality
of evidence. I really enjoyed your discussion on un randomized controlled trials because this quote unquote gold standard of medical studies that might not always be the gold standard, And I was hoping you could tell me a little bit about the times when the true gold standard might not be, for instance, an RCT, but it might be something else entirely, or it might be unethical to do a randomized control trial in that situation.
Yeah, I think we've seen quite a lot of examples where treating it as that kind of cookie cutter, this is the only method we can use coleadisant problems. I mean, smoking cancers are very well known one that we couldn't just have in action because you can't get that level of perfection.
I mean actually even the first.
Randomized control trial in modern medicine, which is ninety forty seven, so stretcht to mycin, a trial for TB.
Austin Bradford Hill, who led.
That, made the point that actually Stretto mycin had some very promising looking lab data and kind of early signals, and he suggested it would have been unethical to withhold it from patients if it was available. But actually as a ninety forty seven there were currency controls. The UK and its post war state couldn't get enough dollars to buy uptimize it. So there wasn't enough to go around. So in that situation, they said it would be ethical
to randomize because there's not enough of it. So there's not enough of it, you might as well randomize and just learn something along the way. And I think we've seen that in other situations. I mean other sort of examples that you see where things are very difficult to randomize. You can think about natural experiments, a lot of the well known moms the Vietnam Draft, where people essentially randomly
assigned to go to war based on their birthdays. A lot of economists have done in Nobrock pires winning work using that to understand the effects of war on subsequent life outcomes, because it's not something where you can fully design that experiment, but you can then make use of
what you have available. So I think a lot of it just comes down to this issue we want to understand called effect, and the benefit of randomization is a lot of the other things that would influence whether or not you know, someone's getting a vaccine and someone's getting a disease, because you're randomizing on the vaccine, on average, those will cancel out as effects, so it gives you
that quite neat benefit. But of course, you've also got the challenge that you might run a population in one group when one population that doesn't generalize to someone else.
You've also got the time issue.
So for diseases that evolve, you know, you might run a trial now against flu or COVID or something a year later that's going to be a different variant, and
to what extent can you carry over those conclusions. I think we see a lot of examples in the literature where, for instance, someone might run a trial in one population for one disease, for flu, for example, and then see a very different result when people look at population patterns elsewhere, because it's a different immunse structure, it's a different strain,
it's a different time period. And yeah, I think we can't just say, well that study from a few years ago as the gold standard, we're only going to use that one. We have to think about how these things move along. I mean that being said, though, I think in COVID there were missed opportunities I think to gather much stronger data. I think it's very hard to justify running those kinds of studies As a threat increases. I think when epidemic's going up, your time to kind of
try and randomize it. I mean, I think essentially countries have to take that threat, as the evidence suggests, but I think particularly as countries lifted measures, that was often just done in quite an ad hoc way, and we
could have done much more kind of staging. In the UK, there were some early studies, for example, of can we use rapid tests so people test themselves every day rather than quarantining for like a week or two, and then in practice a lot of people just didn't bother But apart from that, you know, I think there's a lot of these debates we're still having, and we probably could have got better answers for that with some higher quality studies, so not necessarily even an RCT, just just making use
of what we had with more observations.
One thing that I feel like during the COVID pandemic, especially the early months, was this desire from the public to have the answers. And I feel like there's a lot of variation in how willing someone is to say I don't know. And I'm wondering your feelings on this. Do you feel that scientists in particular have a difficult time so saying that they don't know the answer to something like do we need to embrace uncertainty more in as scientists or do you feel like there's that we
are embracing it but just not communicating it. Well, Yeah, I think.
That's a really good question. It's kind of how I guess how personality in politics and all these things go. So and I think, I mean, there's been good reviews of evidence showing that the overstated certainty just just undermines trust and confidence, whether it's kind vaccine to its other things. So saying yeah, this is one hundred percent, say there's absolutely no risk, and if there's even a tiny risk, you then kind of undermined that.
Yeah. One of the challenges with kind of that over.
Certainty, I think, particularly once you make that public statement, it's very hard to back that, and we saw that with some of the airborne trying some even health organizations say it's not airborne fact. It's very difficult to then walk that back. But I think it's fine line because you don't just want to say we have no idea. You want to try and communicate the way that evidence.
I think some countries did that better than others, particularly them Mark Singapore s pling to mind on their reopening where they said, this is the data we're looking at to do this, that might change and this is kind of how we have to work things through.
But I think one of the one of the.
Difficulty I think because any emergency it goes on for that long, is you know, you have some people who have very loudly said something's one hundred less, one hundred times less severe than it is, and then they're kind of very nailed onto to having to keep promoting that. And I think it is There was one that one of the government advisory committees I sat on. You know, so a lot of the early alpha very and early delta very and a lot of this early severity came
out of this group. And there was a phrase that became used quite a lot, which was tell me why I'm wrong. If you have that discussion where you want to get criticism, if you present stuff and especially people more senior and say is this is this correct, it's very hard for people to kind of come in and say, oh, yeah, actually I've sported a problem, especially if there's a power
dynamics or seniority and other things. So I think there was a lot of that thing where people present work and right, tell me why I'm wrong, tell me what I'm missing. And I think that's quite a healthy attitude in that kind of environment to be much more, you know, looking for weaknesses and being able to.
Kind of lay out.
And I remember, actually I think it was when was it the gamma variant is sort of emerging in Latin America And I gave immediate interview, and when they write it up, it was basically, you know, Dotowski doesn't really know.
It was the kind of open but in that situation, we didn't.
And it is hard to do because I think, you know, especially people asking you questions around your area of expertise, I think in terms of how to balance that not just saying I don't know, but saying, well, we do
know this, and we can make some judgment. And there was this wonderful study in the nineteen fifty one is by the CIA analyst, and it was about words we use when we're unsure and words about judgment, and it basically realized that people used probable and possible to meet all sorts of things, and they all know had kind
of different notions. And he said, yeah, humans will go out of their way to making a judgment about something that we'll often you know, the risk is you get the uncertainty where we're like very hazy, like, oh it's you know, it's a definite possibility. And actually, in some cases, like with you know, if you've got an emerging threat and you've got experts, you do actually want them to put a number on it, you know, even if there's uncertainty, you want them to say, I am sixty percent sure
that this is the case. And there's been a lot of nice work, you know, even around things like super forecasting, where people make those predictions and you can go back and then look because you know, if people are well calibrated in their uncertainty. Yeah, if you say you're fifty percent sure about a list of things, about fifty percent at the time those things should happen. So about half the things on that list should occur. So there are these situations where I think we can get better just
about thinking about our own uncertainty. And one of the things that I actually tried to do, I've tried to do a lot of kind of emerging threats is even just just writing down what you think is going to happen. Because I think we're great, you know, the human mind
at like kind of rationalizing. Oh yeah, maybe I did think that, And so yeah, I did quite quite a lot of like where where you could state I actually think the vaccine is going to be pretty good, or you know, I think this, and like this is where social media, when it was maybe slightly less polarized, is quite helpful because you could just put a post out.
And I think I was always very careful. I didn't delete any of my tweets during COVID because I was like, I actually want that record, and there were time I got one. You know, I was in Singapore in feb twenty twenty and their policy was don't wear a mask unless you have symptoms, and I think I tweeted I was like, yeah, that seems like a sensible policy, and that seems quite at evidence space. And now we'd probably with some of the studies not look back on that
as being the best post. But so yeah, I think it's almost that as well as overstated certainty. I think it's also holding ourselves to account, even IF's just you know, privately about how confident we were and what played out.
I want to close out by asking you about the subtitle of your book, which is the art and science of certainty? And I want to know about the art part of this, what is the art s?
So I think for me it was the more I dug into this, the more I saw these other elements beyond kind of pure logic, pure observation coming in. I mean, even if you look at what was essentially a bit of a mathematical civil war in the late nineteenth century, where a lot of these ancient Greek theorems, you know, things about the properties of triangles started to break down because people started to draw shapes on spheres and other structures and come up with functions that these supposedly proven
theorems didn't hold. And I think one of the reasons that was really controversial was there was this idea that there's a universal truth out there about the world, and actually in this situation, it kind of depended on what assumptions humans were making and what we were willing to kind of define.
And even in this.
Supposedly pure subjects, there's still these debates around well, it kind of depends on which one you want to pick, and that will change the answer. I think even in science, it's a lot of these situations where you know, we can accumulate the evidence, but then you have disagreement about where you set the freshal. I mean that this kind of five percent cut off has become very popular, this sort of p value that the chance that you'd get a result that extreme if there was nothing going on
or you're no, no hypothesis was wrong. But that was kind of arbitrary. I mean it was partly picked just for convenience, that this was you one hundred years ago. The calculations just a bit easier if they picked a value one fished a lot to work, just easier to pick a value around point zero five. And others who were more pragmatic, you know, working in business on something and thinking, well, actually the evidence is a bit weaker, but that's still useful to it. So there's this kind
of human balancing act. And yeah, we saw it again, things like legal cases where how much you value different types of errors depends a lot on the individuals. I mean, one of the examples that that fan fascinating in the book was Einstein, when you moved to the US, got very angry about peer review because he sent some to send something to a journal and it came back as like, oh, we've got another opinion on it, and he was like whoaah whoa, Like, why haven't you.
Just accepted my work?
And actually Max Max Plank, who published some of his like amazing early papers, Plank made that point that actually, I would rather kind of publish a few things that are a bit you know, nonsense, than this is me paraphrasing the miss a really important idea. So for him, his threshold was like, I want to set the threshold low. Admittedly mainly amongst kind of physicists he knew, because I don't want to set it too high and miss a good idea, and.
I think we all we all have those.
This kind of that's where the art, I think creeps in that that kind of subjectivity in not just the evidence, I think one of For me, the real difference was something like proof is it's not just generating data. It's how that data interacts with the world and the decisions we make. And I think that's where things get really interesting. It's like where do we actually set the bar for evidence and then both to convince ourselves but then go out and convince others too well.
Professor Kocharski, thank you so much for joining me today. This This was such an enlightening conversation, and I really did. I loved your book Proof, so I appreciate you coming onto the show.
Thanks great to talk.
A big thank you again to doctor Adam Kocharski for taking the time to chat with me. If you enjoyed today's episode and would like to learn more, check out our website This podcast will Kill You dot com, where I'll post a link to where you can find Proof, the Art and Science of Certainty, as well as a link to DODR. Kocharski's website where you can also find his other book, The Rules of Contagion, Why Things Spread
and Why They Stop and Don't Forget. You can also check out our website for all sorts of other cool things, including but not limited to, transcripts, quarantine recipes, show notes, and references for all of our episodes, links to merch our bookshop dot Org, affiliate account, our Goodreads list, a first hand account, form, and music by Bloodmobile. Speaking of which, thank you to Bloodmobile for providing the music for this
episode and all of our episodes. Thank you to Leona Squilacci and Tom Bryfocal for our audio mixing, and thanks to you listeners for listening. I hope you liked this episode and our loving still being part of the TPWKY book Club. A special thank you, as always to our fantastic patrons. We appreciate your support so very much. Well, until next time, keep washing those hands, FU