The Worm Wars

Give me two seconds. I need to close the windows. I was like, it's so fucking hot in here, and I was like, I'll try to do it with the windows open, but there's like shitloads of bird sounds. Yeah, totally. Alright, no birds! This is now a no birds podcast. You Have never sounded more like Jerry Stiller. No birds. No birds. Um What do you got? What have you got for us? Uh hi everybody and welcome to Maintenance Phase, the podcast that's

Crawling out onto the sidewalk after a nice long rain. Oh, it's a worm. It's a worm opener. All I had was worms. You had two words, worm wars. Sure, man, I'll take my inspiration from my morning walk the other day. I like a hug. I'm Aubrey Gordon. If you would like to support the show, you can do that at patreon.com slash maintenance phase.

And if you want to just keep listening, just keep listening. We're so happy to have you. You're here with us. Today we're talking about worms. Yes. So, okay, I'm so excited. Because I like telling you stories and I like fucking with you. And so I I told you I I told you nothing other than the name of this episode, Worm Wars. This is the controversy that tore methodology Twitter aside.

In 2015. And I want to tell you all about it. Tell me. So the hardest part of this episode is gonna be getting through the first five minutes. Oh. There's just some like gross stuff like body horror that we have to get through. Okay. Are you familiar with intestinal worms? Uh I mean not really. Okay. So I assume they're like parasites, yeah? Yes. Okay. So

Somewhere around two billion people in the world have intestinal worms, which are like exactly what they sound like. They are little tiny worms that live inside of your digestive system. Okay. Some intestinal worms have lifespans of like one to two years. And they'll literally just like

attach themselves to the inside of your intestine and just like be there for years and then they die. Okay. Most people have light to moderate infections which don't really have any symptoms, but sometimes you'll get like much more heavy infections. And so this causes various symptoms. It it it usually happens through giving you anemia. It makes you tired, it gives you brain fog, you have to sleep all the time.

It can stunt kids' growth in the developing world. It can make them too skinny. I have never had a worm infection or parasite or any of that kind of stuff, but I have been anemic before, and boy oh boy.

It doesn't seem like it's gonna be that rough and then it's pretty rough. Dude, a friend of mine passed out in a Starbucks because she got anemia. It's just you just can't get your energy up, like at all. Yes. So okay, so here's the really gross part. So the way that worms spread is you get these worms in your intestine lining and they lay eggs. They lay like hundreds of thousands of like little microscopic eggs. And these eggs end up in your poo and then you poo out the eggs.

And so the reason why worm infections are endemic throughout the developing world is because oftentimes there isn't running water. So in some areas people will kind of use soil to wash their hands after they go to the bathroom. They'll just kind of grab some like dry soil from the ground. Sometimes they use poo as like fertilizers in gardens.

And then it ends up in their food, or it's like they're walking around and they have a cut on their foot and there's like a little tiny egg that like ends up getting into the cut and then like swims its way to their intestinal tract. Sure, sure, sure. So it's just like the poop human interface. is like how people end up getting worms. And it's like this rolling snowball where like the more worms there are

in the soil, the more likely kids are to get it, the more likely kids are to spread it, etcetera. It used to actually be a really big problem in America. Here is what I appreciate about this whole episode is I didn't have to bring up anything related to poop. I know. You just fully picked a topic and you were like, you know what I could go for right now is a little poop research. The thing is, I I saw a methodology fight.

And I was like, Oh, it's gonna drive me into the poop world, isn't it? I ha I have to go inward. So the protagonist for this episode is a guy named Michael Kramer, who is currently a Harvard professor. Okay. Before all of that. He had spent some time in Kenya as a kid, and he was really fascinated by like why are people poor in this country? Like, what are we doing to alleviate global poverty? He eventually gets his degree in economics. And so he decides to do his graduate work in Kenya.

So he moves to Kenya and he starts befriending people in the development sector there. He meets like World Bank people and NGO people and charity people. And what he finds is that the field of international development is like in kind of what other people will later call a statistical dark ages.

This is like the early nineteen nineties and the way that people do development, it's kind of on like the live aid model, right? Where it's like a a celebrity does something and there's like this big fanfare and then like no one ever follows up afterwards.

Yeah. And if there's any like statistical rigor or like studies being done, they're oftentimes before and after studies. So they're like, We gave everyone a goat and then a year later they were doing better. And it's like, Well, was just everyone doing better in the country that year because there was like a really good harvest?

or something. Like you're not you're not comparing them against a control group. Or was there another policy intervention? Right. There's like so many things that it could be. Right. Think about the last year in your own life. And think about if someone from the outside tried to determine what was the cause of any changes in your life, right? Like that would be kind of a fool's errand, right? Right.

And a lot of the projects are done in this sort of like common sense way. This is oftentimes how you get bad development projects where people are like, Oh, it's just common sense to you know, give everyone a bicycle. Or whatever. It's like, well, did you ask them first if they needed a bicycle? Like if you give everybody in the village a bicycle, is that gonna like change the dynamics of the village? Like it it's just being done in this like really unsophisticated way.

Yeah. So Michael Kramer has a friend who works at a Dutch NGO and they're about to implement a program where they give kids in Kenya textbooks. So at the time, there's like 30 kids in every classroom and there's one textbook per classroom.

So again, common sense. It's like, well, if everybody had a textbook, this is obviously gonna make like their grades better and like school performance, attendance, everything's gonna get better if everybody has their own textbook. Obvious, right? So Michael Kramer suggests to his friend, he's like, No, no, no, don't do a before and after study.

Instead of just giving this to seven schools, take fourteen schools and give seven of them textbooks and the other don't give them textbooks, and then you can actually compare the effect of the textbook. So they do this, the project goes for a couple years, and lo and behold, there's essentially no effect of the text.

Uh-huh. So after the textbooks don't work, there's this period of kind of testing stuff, because it's like, well, that's weird that the textbooks don't work. So then they try like chalkboards, like a lot of the schools don't have chalkboards. So again, they split into two groups. Turns out that doesn't work. They then do a thing where they hire more teachers at half of the schools. That doesn't work.

So it's like they're kind of going over these sort of obvious quote unquote education interventions. Mm-hmm. And it turns out that like education interventions are not working to improve education. They're like, hmm.

So one night Michael Kramer is talking to a friend of his who works at the World Bank. And he explains the symptoms of worm infection to Michael Kramer and he tells him that like there are treatments available for worms, like there are pills for basically every kind of species that's endemic in that part of Kenya. And like they're really cheap. and diagnosing the kids.

is actually like really expensive'cause you literally like you literally have to go through their poop and like measure how many eggs there are. No. And so it's such a pain to diagnose this and it's so easy to give the kids these medical treatments, like they don't really have any major side effects.

They're pretty cheap. They're already available in that part of Africa. And he's like, I wonder what would happen if we just like gave all the kids these deworming tablets. Because some of them are probably infected with worms and like just don't know it. So they do the same thing. They find a district with 35,000 students. Seventy five schools.

And they do the same thing. They separate it into control groups and treatment groups. They roll out this mass deworming program where they just give every kid deworming pills, right? Okay. So I'm going to send you an excerpt. This is from a book called Doing Good Better How Effective Altruism Can Help You Make a Difference by William McCaskill. Quote Kramer did an experiment to see whether treating children for these intestinal worms had an impact on education.

The results were striking. Absenteeism is a chronic problem in schools in Kenya, and deworming reduced it by 25%. In fact, every child treated spent an extra two weeks in school, and every$100 spent on the program provided a total of 10 years of additional school attendance among all students. Enabling a child to spend an extra day in school therefore cost just five cents. It wasn't merely that deworming children quote unquote worked at getting children into school.

It worked incredibly well. It worked incredibly well. I mean, fucking yeah. Yeah. If you have something uncomfortable to painful to debilitating going on with your body. When you feel better, I bet you do better in school. Yeah. So I mean it's it's a very appealing story also because

They've tried all of these educational interventions, and it turns out that there's this sort of like zigzag solution. That maybe it's not an education intervention. Maybe it's like there's something broader and bigger at play that is just invisible to people that are in the school and also development aid workers. Yep, totally. So the main headline from this study is that they reduced absenteeism by 25%. That's the thing that ends up in William McCaskill's book.

That's like the kind of the main thing, it's in the abstract. But there's also some other effects. One of them is like there's health benefits, so the kids who got the deworming were slightly taller, they're slightly heavier. Mm-hmm It was actually nice to read a study where it's like the kids are fat. Yeah. Yay. It's like it's like just once. What a win! They also had lower rates of anemia. And one of the really interesting effects of the study is these spillover effects.

So because worm infections are an infectious disease, right? One person gives it to another person. Once you blast hundreds of kids in these schools with deworming pills they then don't spread it to other people. So what they find is that there's there's actually attendance effects in schools like three kilometers away and six kilometers away. There's all these like other benefits that come along with basically just like breaking the chain of infection. And they also find long-term effects.

This is from McCaskill's books. When Kramer's colleagues followed up with the children ten years later, those who had been dewormed were working an extra three point four hours per week and earning an extra twenty percent of income compared to those who had not been dewormed. In fact, deworming was such a powerful program that it paid for itself through increased tax revenue. Wow. So it's like you can then even like fast forward ten years.

And you find that like the dewormed kids are doing better. Big if true. Big if true. Aubrey, it's the first five minutes of the episode. Uh you know everything I tell you at this point is true. Everything goes according to plan. The nice white guy from Harvard is definitely right, and no harm is going to come from his actions. Stop remembering the way that I tell you stories. Um

Stop remembering the premise of our show. The premise of our show. Uh so over the next decade, basically, deworming becomes like the hottest. idea in international development. I was working in international development at this time. I remember all of like the deworming national plans that were coming out. It paralleled the rise of this effective altruism. thinking, where in Michael Kramer's original paper, which was published in two thousand four,

A lot of the paper is actually about like the cost benefits of this, right? That, you know, when it comes to something like hiring more teachers. Well like every single teacher costs a lot of money and there's, you know, there's payroll and there's all these other costs associated with these like educational interventions. Whereas this one, like the deworming pills cost thirty cents each. The the benefit of this versus the cost of doing it.

It's just like unprecedented in international development. So in 2007, Michael Kramer founds a NGO called Deworm the World. Okay. Which like helps governments implement their like national deworming plans.

There's also this charity Give Well, which basically has this model of like we're gonna look at all the evidence and we're just gonna tell you that like if you have a hundred bucks to give away, here's probably the most effective thing to give it to, you know, if you if you wanna help people, this is what you should do. And that's like one of the few charities that is ranked by Givewell is this Deworm the World Initiative and other kind of deworming charities.

There was like a dewarming strategy by the WHO. The World Bank was really big on dewarming. It was like dewarming was like the hotness, right? Kenya has a national plan on the basis of this two thousand four paper. India declares February 10th National Deworming Day and has a program to deworm 140 million kids a year. Whoa. This is huge. According to the Gates Foundation, at one point they put out a paper calling this the largest

Public health program ever attempted. Wow. Huge full court, like decade-long full court press on deworming. There's part of this that feels very like Ted talkie life hacky. This one simple trick Aubrey. Fixes everything. Aubrey. Am I getting ahead of myself? I love that you're at the point with the show where you're like, wait a minute, there's a good story here. It's probably fake. I don't trust it. Like, mmm.

So okay, I'm about to send you a couple paragraphs from a Guardian article in twenty fifteen. That kicks off The Worm Wars. Whoa. This is one of the longest excerpts that we've ever read on the show, but I wanted to get I wanted to you to get like the flavor of this. Great. The headline says New Research Debunks Merits of Global Deworming Programs.

And I say fucking called it. Twistity twist. Quote Deworming Children, once ranked by Nobel laureates as the fourth most effective intervention to solve the health problems of the whole world offers very little benefit despite the millions of dollars spent on it, according to a reanalysis of the evidence. The World Bank and the UK government are among the big funders of deworming programs that were claimed to improve not only children's health and growth.

but also their educational attainment and ultimately their country's economic prosperity. However, much of that optimism rests on one particular study carried out in Kenya in nineteen ninety-eight to nineteen ninety-nine. which reported that deworming improved attendance not only in schools where children were treated and taught about avoiding worms, but also in other schools because of an apparent reduction in transmission.

Researchers from the London School of Hygiene and Tropical Medicine have now reanalyzed the trial data and published two papers in the International Journal of Epidemiology. Researchers used methods common to health research rather than those used in economic research, as the original paper had. God damn it. Good sign. The epidemiologists finally look into it. Ten years later. There had been no increased attendance in the schools where children had not been treated, as the original study said.

That was down to calculation errors and was wrong. The researcher also found missing data. And some evidence of bias in the finding that School attendance improved. Shocking. Who could have seen that coming? What what is your what is your overall impression? What is what is the plain language paraphrase? Of what you've just read. The fucking epidemiologists never looked at an epidemiology issue, and it was studied by fucking economists.

What are we doing? And then like whole fucking nations built their interventions around what a fucking economist thought was good for public health. At this point in twenty fifteen, I've seen various numbers going around. One of the numbers is that one billion deworming pills are being distributed annually. What? Under programs like this. I assume you're gonna walk me through this. Yes. It sounds like what they have debunked here is the idea that school attendance

did not improve in schools where kids were not treated. That's a very good catch. Yes. It's not the whole thing is garbage. It's just This idea of school attendance going up in other schools is not true. That's what I'm taking away from this paragraph. That was a very good read. That like a lot of people did not notice at the time. You just notice now. Oh fun. So now we get misunderstandings. So this is the thing. So we had to talk about what they actually did and what they actually found.

So as you know, there's like in psychology and other sciences there's now this like this thing called the replication crisis. Uh-huh. Essentially the idea is that like there's all these canonical studies that have affected policy and have affected public understandings. And they're basically rerunning them.

And in a lot of cases they don't pan out. So things like the the infamous like marshmallow test. Mm-hmm. What happened here is that people looked at this study from 2004 in Kenya being used to justify deworming 140 million children in India and we're like, wait a minute, like a lot is resting on this paper. Let's just look into the findings. So like they didn't redo any studies. This wasn't a replication.

What they did was they went to the original author, so it was Michael Kramer and his co authors on the original paper, and they said, give us all your data. Like give us the software that you ran, give us the original, like the questionnaires you used in the school, like just give us everything and all we're gonna do is just run it again. Okay. So to their great credit. Kramer and his co authors like give them everything. Okay. So these reanalysis researchers spend months.

going over everything and basically publish like a reanalysis of this paper. So the biggest thing that they find is there's data missing and there's data that's miscalculated. You know those like spreadsheets have all these conditionalities where like one cell refers to another cell refers to another cell? It's like they basically got one of these little codes wrong, and that ends up cascading down to all of the other codes in this way that was like invisible to the original research.

So it's essentially like the level of like a transcription error. Exactly. So as a result of all of these errors, everything of of of kind of recalculating the data, all of the health effects of deworming disappear. What? The anemia link is gone, the kids difference in heights gone, the difference in weights gone. Wow. And then essentially all these spillover effects. So like all the stuff about like the indirect three kilometers, six kilometers.

Those effects, the the six kilometer effects are gone and the three kilometer effects are much, much, much weaker. To me, the main thing that comes out of this statistical reanalysis I mean I mean this is something very well known in studies of school attendance in sub Saharan Africa and really throughout the developing world.

that like if if you rely on school registers, they're not very accurate. Teachers write kids down as being here when they're not really here. So if you want to do a study that actually measures school attendance, you have to send people to the school to like physically count the kids.

Which to their credit they did in this study. Like they sent people to like, okay, one, two, three, four, five in every school when they wanted to do it. And they did it at random times, right?'Cause you want to get an actual like an accurate ton of the kids. What they found in this reanalysis was that the deworming schools got way more visits. From the counters, from the auditors, and the schools that got visited more had better attendance.

It i it could be just the fact that someone was checking that's doing this and not actually deworming. Yeah. So this is this this is what turned me into the fucking joker on this.

Once there's all this like statistical stuff going on and this extra scrutiny on the original paper, people also go back and just like read the original paper. The first thing that jumps out at you is that this wasn't a deworming program. It was a deworming and education program. So the schools that got the deworming pills also got like classes about like how why it's important to wash your hands, like better sanitation techniques, how to prepare food.

The kids and their parents, in some cases, were getting information along with the deworming pills that was related to not getting reinfected with worms. Nah, but it's definitely those pills. But it's definitely I mean this is the thing is like The the the researchers say

that i when you survey the kids about like how important is it to wash your hands and stuff, there's actually no difference between the kids that got the education stuff and the kids that didn't. Wow. As we talk about on this show all the fucking time, like public education campaigns don't really work.

Yeah. Knowing that washing your hands is good doesn't give you running water in your house. Yeah. The researchers are like, oh well, it doesn't matter that we also gave them education programs because the education programs didn't work. So we can scale up the pills without scaling up the education. But the thing is they didn't check the effect of the education on the school attendant. There's literally no way to separate those things. You're doing two things at once.

And then school attendance gets better. I am Just really breathing through feeling my blood pressure rising. You don't have to be like the world's greatest fucking scientist to figure out that if you give somebody deworming pills and also like a bunch of like educational resources and support for their lives.

You can't then say everything that's changed is just because of those pills. We've been through this so many times as a field. This is as a former development aid guy, this is what drives me nuts, is that what you have over and over again

is you have a technical intervention, something like a medical intervention or like better lighting in schools or whatever, something that's easy to scale, right? But then you also have all these social things that go around it, right? Like getting to know the kids, having them attend these educational seminars, having probably just like more adults in the school generally if they're participating in a study, right?

And then it's like when it's time to upscale, we're like, oh, this is a proven development intervention, so we're gonna upscale the technical part. Yeah. But you're not gonna upscale all the fucking social stuff that made the technical stuff work. God damn it. That's not the purpose of a randomized controlled trial. The entire point of all of this.

was to be like, we're gonna do it like pharmaceuticals do it, right? We're gonna do it in a way that allows us to say with certainty that like this is the intervention that mattered. Yeah. And then they do this like one oh one shit. Where it's like, oh well we're gonna do two things at once and then upscale one of them. Yeah, totally. And also like this does feel sort of related to because this is all happening at the site of

Education as a field. It also feels totally fucking wack-adoo and deeply predictable to have a bunch of economists roll in and be like, Teachers, you've been fucking it up. You're welcome. We're here. We figured it out. But then okay, are you are you ready to explode? Oh no. I guess. So the other thing that people start like bringing up about the original paper is that that number of that it it reduced absenteeism by twenty-five percent.

Okay. school with like a ninety-eight percent attendance rate. And then you switch the vending machines from like Coke to Pepsi. Sure. And then they have a 99% attendance rate. You could say that they reduced absenteeism by 50%. Oh god damn it. That sounds really impressive, but what you've done is you've gone from 98% to 99%, which is not that impressive.

No, this is the like Like fucking diet marketing of like you're twice as likely to lose weight on our diet than on this other diet, and you're like, Well, okay, so you lose one pound instead of half a pound in a year. You're twice as likely to win the lottery when you buy two lottery tickets. That doesn't mean you're gonna win the lottery. The actual attendance rates at the schools that didn't get deworming pills. The attendance rates are seventy five percent.

At the schools that did get deworming, it's 82%. Oh god. Is that nothing? No. Like it's a 7% bump in attendance rates. But also a 7% increase in attendance. Much less like we must scale this to 140 million children immediately. Uh-huh. Another thing that again is in the paper that was published in 2004. is that the dewarming pills did not have any effect. On cognitive ability or school performance.

So nothing. Nothing. So exam scores, they did some like IQ testy pseudoscience bullshit on the kids. Nothing. So it only affected Attendance. I feel like I'm resorting to fucking Lama's breathing to get through this shit. Like I feel like I'm this close to like To like deal with my fucking rage about this sh well the whole the whole narrative of this intervention doesn't really make sense.

Yeah. The the theory of change that deworming is supposed to do, right? Is like you give kids the deworming pills, their health improves, their school attendance improves, their test scores improve, and then their life gets better, right? That's like the chain of events. Right? Yep. But what we have now is we have you give the kids deworming pills.

It doesn't seem to affect their health. It does affect their school attendance, but doesn't affect their school performance. Yeah. It just kind of a mess. I don't know, man. This feels like an indicator of the depths of two major appetites. One is

how fucking hungry we are at all times for a silver bullet thing. Mm-hmm. And how much more fun it gets to people and exciting when it's like an unexpected thing. Like Oh, if you get this one pill, you get better at school. Imagine. Totally. And Appetite number two that is sort of shaping this is the unending appetite of white people.

To be like, we found the thing, we solved it, you're welcome, you couldn't do it yourself, we did it for you. Exactly. Not a bad impulse to want to help, a terrible fucking impulse to want to like save people from themselves. I actually got radicalized on this I think a decade ago now where I was writing an article about like how big ideas were ruining international development. I called up

One of the charities that was doing one of these projects in India. They were they were expanding the deworming, mass deworming programs to India. In one of the most Kafka esque interviews I've ever had. I was asking them like, okay, how's the project going so far? And they're like, It's great. We've we've distributed ten million pills or whatever it was. And I was like, okay, great, like what are the results on school attendance?

And they were like, Oh, we're not measuring that. What oh because it's it's proven. Like it's proven that it affects school attendance. So we we don't really have to measure that anymore. And I was like, Well, it's proven in seventy five schools in Kenya ten years ago.

You're doing this in India, which is a completely different educational context of schooling and like the government's role in school, everything else. And like so you're just not testing the effects anymore and they're like, But we are testing the effects. Well then what are the effects? They're like, Well, we've distributed ten million pills. That's okay.

Okay, so you and I get into this shit often. That's a fucking output. It's not a fucking outcome. Exactly. That's step one to do what you set out to do. Like it's truly fucking wild. how often you and I get into data on this show and the results of a given study or a clinical trial or a whatever are just here's what we did And then that celebrated like uh something that has like an impact on people's health out in the world. But Aubrey saves

Save your rage. I'm getting too angry too soon, aren't I? This is a three-part debunking and that was only part one. What? That is frankly nothing short of a personal attack. So At the same time that this reanalysis of the two thousand four paper is going on, I think this is a coincidence.

There's also a meta-analysis of every previous study on deworming that's ever been done. Oh. One thing that drives me absolutely batshit about this story and about the way that it's presented in this like effective altruism book.

is everyone pretends like no one thought of fucking deworming before. There have been deworming projects since the nineteen seventies. Yeah. So in twenty fifteen there's a Cochrane review of every previous, like decent study on this. It it it covers forty five Studies going back to the late nineteen seventies. It includes 20 randomized control trials in Africa.

It finds the review says that There is quite substantial evidence that deworming programs do not show benefit in terms of average nutritional status. Hemoglobin, cognition, school performance, or mortality. Nothing That's the good stuff right there. The world's largest randomized control trial, or at least according to the study itself. was performed in India in twenty thirteen on deworming, it finds no effect.

There's a huge randomized control trial in China, which also does not find anything. There have been ten years of various deworming projects in Myanmar. and the rates of worm infection are now higher than they were ten years ago. What Which I d I don't blame the I don't think that has anything to do with the studies itself. I think again

It's like I think there's something else going on there. Sure. What drives me nuts about this is like, why would we still be talking about this two thousand four study in Kenya when there's like Fifty other fucking t tests of this by now. Yeah. People have tested this. And it has been found wanting Time and time and time and time again. It doesn't have any of the effects that people said that it had. And that's on any measure.

Right. We're no longer in school attendance world. I mean they they mention school attendance. This is actually like one of the main things of like the the worm wars, like the big methodological debate that comes out is

Because Michael Kramer and others say that like you can't test whether it affects school attendance and like a meta analysis'cause there's not enough studies that do s school attendance. It's only based on this one and they all like they throw accusations back and forth, blah blah blah. But it's like there are other studies that look at school attendance and it doesn't seem to have any impact on it. And also Even if it did, it's like, why are we talking about school attendance?

When like it doesn't affect school performance. Right. And it doesn't affect health. Right. Then what? Then what? You got'em in the room, then what? And also there's also some like interesting sort of second order effects that oftentimes when school attendance rates go up in the developing world the teachers are then overcrowded and like like lots of the kids end up doing worse on tests because they're getting less attention from their teachers.

Not to say that like school attendance is bad or whatever, but it's like anytime you're doing these development projects, there's always these weird second order effects. And so it's like it's not there are actually places where even if it did increase school attendance, if it increases it significantly and they're not able to hire teachers, it might not actually have the long-term effects that you're going for. Yeah. Every study on this

seems to find that it doesn't really do what they set out to do. I in a lot of these studies You can tell they don't want to admit. that it didn't have any effect. You and me are like trained to say this. Now you'll read the abstract and they'll be like, we distributed 50 million pills and then like we surveyed a bunch of kids and the kids had heard of the program. Yeah. And you're like, wait.

That's not like no one would look at a deworming program and be like, have the kids heard of this or not? Yeah. It's like you're clearly picking something you can publish.

We know that calorie labeling works because some people saw a billboard about it one time. Exactly. I was subtweeting this when uh when we were recording that episode. I was subtweeting the deworming people. I have to say I really appreciate how often in the researching of an episode you or I end up like cryptically tweeting about something so that we don't spoil it.

But only for the other person? The thing is, I was gonna block you on Twitter because I was like about to tweet something salty about deworming and I was like, I don't want I had a whole plan. I was gonna be like, Aubrey, I just want you to know I blocked you on Twitter. It's not you. I I tweeted.

But then I was like, Oh, I don't want her to know. So then I just I I walked away from my computer ultimately. Here's the hilarious thing. As you know, I have been very off Twitter for the last like, I don't know, four months or so. So the thing that is hilarious to me is

I think the rest of the internet would be more likely to figure that out than I would. Yeah, exactly. I know. And then did you know Mike blocked you on Twitter? Yeah, it's like Mike blocked Hombre on Twitter. Tremble in Paradise. So okay. final stage of the debunking. Okay. Okay, we've talked about the 2004 paper problems. We've talked about the field of deworming programs in general, meta-analyses problem.

We're now talking about the problems of upscaling. Okay. Because this is another thing that as a former development guy, I know that what Happens in seventy five schools in Kenya doesn't happen the same way once you start rolling this out on like tens of millions of kids in like in entire countries. Sure. So this is something that happens in two thousand eight in a region of Tanzania called Morogoro.

Tanzania adopted like a national dewarming strategy and was rolling this out across the country. And so the idea was that they were gonna give kids a bunch of deworming pills. And they had a town hall meeting a couple days before this happened to like tell the parents about it. But according to the parents later, they didn't like really get the message that this was happening or like they just had other things happening that day, like agricultural duties, things at night that they had to do.

So very few parents ended up attending this like one town hall meeting being like, we're gonna give your kids deworming treatments. And according to the parents who actually went, the information that they gave the parents there was not even very good because they they didn't really explain to the parents that they were doing prophylactic deworming.

You know, deworming pills are like pretty widely available in sub Saharan Africa and like when you get symptoms of a worm infection, like you go to the pharmacy and if you can afford it, you buy the pills and you take the pills and the the worms are gone. These are known these are like known drug treatments. But the idea of we're gonna give every kid deworming pills, like just in case, that hadn't really been done before.

It's a little bit like some teacher coming to you and saying, like, well, you know, your kid takes Tylenol for headaches, right? Well, we're just gonna give every kid Tylenol at 3 p.m. every day. You'd be like, What? Right? Like it just doesn't really like fit with your understanding of like how that particular medication works. Yeah. So after this town hall meeting that like very few people attend and doesn't give very useful information anyway, they start doing this deworming with the kids.

But another change that they make is that the deworming pills that people in Tanzania are familiar with, I think it's called albendazole. It's like a little tiny pill, and apparently it tastes kind of sweet.

Mm-hmm. And the kids are used to taking it. You know how it's like kinda difficult to get kids to take pills in the first place? Sure, absolutely. But like because it's small, because it's easy to swallow, the kids are kinda used to it, so it's not that big of a deal to get kids to take this. But there's different species of intestinal worms. So they're also having these kids take this other pill that is like big and bitter, and they have to test the kid's height.

Before they give it to them to to get the dose. And so some of the kids are taking three of this, some of the kids are taking two of this. It's just like big and like really unpleasant to swallow. And the other thing, I cannot fucking believe this, but this second pill that they're giving people, you you should take it on a full stomach. Okay. Otherwise it starts to give you these side effects. But like we are in a remote

very poor region of Africa. Like most of these kids are only eating one meal a day. So it's like they're not taking it on a full stomach. Like whether you told the parents that or not, they're not taking it on a full stomach. What are the side effects? It's like us it's like stomachache, nausea, it's nothing serious.

But the kids start coming home from school and they say, Oh, they gave us all these pills today. They gave us this normal pill, but they gave us this new pill and they measured our height, and Tanzanians are used to their weight being used. To determine uh doses of like various anti malarials and stuff.

Sure, measuring our height instead of our weight. They gave us this new pill instead of this old pill. And also like I have a stomachache and like maybe I've been vomiting, maybe I have a headache. I have these like weird symptoms. Yeah. So the parents obviously are like What the fuck? Like a lot of them have not heard of this at all. Yeah, totally. And another thing that happens is fairly shortly thereafter, a little girl dies.

She dies of malaria, she dies in the hospital, she was not given the pills. But it starts to look to people like these two things are connected, right? All of a sudden these kids are taking these weird pills in school. They're getting And so people on the local talk radio stations start talking about, you know, three or four kids have died because of the pills. Yeah. And so what starts happening is the parents form these kind of mobs and start showing up at the school.

So this is an excerpt from a PhD dissertation that's written about this. It says In some schools the situation turned violent, as parents and guardians argued with teachers who had been administering drugs. In one primary school in Chemwino, a classroom wall was pulled down, school property was destroyed, the national flag removed, rocks were thrown at the teachers, and one was severely beaten. Holy shit!

The list of injuries suffered by teachers included a broken arm, a broken finger, and a serious head injury. In one school, a health worker had the drug stuffed down her throat, causing her to choke and vomit. Holy hell. I know. The riot police were mobilized and dispersed angry crowds at a number of schools using tear gas.

The Ministry of Health and Social Welfare officially suspended the program in Moragoro and throughout mainland Tanzania the same day. So like I wanna be very clear that like there's a lot of really fucking ugly tropes about like savage Africans and like Africans blowing like There's nothing unique. To African parents here, we are talking about a program that is giving a medical treatment, a novel medical treatment, to children without the knowledge or consent of their parents.

People anywhere would have I think very rightfully like freaked out about this. Right. And also like white people in the US are losing their shit about their kids have to talk about race sometimes. Well, yeah, exactly. It is To me, a deeply understandable reaction that like you find out from your kid that they've been given a new pill at school. You don't know what that was. You don't feel like you were notified about it and now your kid is like getting sick as a result of that pill, right?

I don't know, man. Like this is like every episode of like Law and Order SVU. If anybody ever tried that with my kid, they would have hell to pet. You know what I mean? Like all that kind of shit, right? Like This is a way that we talk about parenting and it's a way that a lot of parents feel, which is like, do not fuck with my kids. Also, like, as soon as we start talking about whether a development intervention works or doesn't work, we're already in the wrong place.

Mm. Something can work and not be rolled out well. Yeah. It can work in one place and not work in another place based on like really, really obvious like social and political conditions. So like one of the things that happens after this in Tanzania in 2008 is the ruling party starts blaming the opposition party and says, Oh, they're the ones that started the rumors, and then they use it as an excuse for a crackdown. Yeah. So it's like you might want

Public health not to take place in a political environment? But it does. It just does. That's right. I mean it's i this is by far the worst case of like deworming being rolled out too fast that I was able to find in the world. But also like these kinds of problems. happen pretty frequently with deworming programs because they're being rolled out so quickly.

So, one of the things that they don't really mention in the follow up literature to this 2004 paper in Kenya is that in a lot of schools, The teachers aren't distributing the pills. So you can do these nasty worming programs where it's like, uh, eighty million kids or whatever. You give a thousand pills to the teachers and they just don't give'em to the kids because the teachers don't know why the fuck they're doing this.

They're not getting paid extra to do it. They're not given the context for it. And then you can go back to your donors and be like, we distributed a thousand pills in this rural village. But it's like, did they get to the kids? There's various studies that that measure like the uptake. And some of them it's like twenty percent. Jesus. And like even in this this drives me nuts. Even in the original two thousand four study in Kenya, again, in the paper itself.

The first they they gave kids a couple rounds of pills. Like it was like every six months they gave them deworming. The first time they gave the kids deworming, 70% of the kids that were eligible took it. By the fourth time they're giving kids the pills, only forty-four percent of the kids are taking them. And all of the stuff about costs. You know, the the pills are only like 15 cents or like you hear different numbers. I've heard two cents, I've heard 45 cents.

The original 2004 paper says it cost about a buck fifty per student. Once you add up that like we had to give it to them every six months and then all this distributional stuff. The the low cost of this intervention is based almost entirely on like pharmaceutical companies giving it away for free. A lot of like volunteer labor to roll this out, teachers doing this with like no sensitization. All of like the low cost justification for these programs depends on the kind of like fast

Cheap universal rollout that caused the problems in Tanzania. Yeah. As soon as you start slowing down to actually see if this is the right thing to do. The cost savings often disappear. Well, and also like this feels like very sort of like life hack kind of adjacent, right? Yes. It feels like doing that, but with like

entrenched public policy and public health issues. Right. Like if someone came along and said that shit about like Brexit, they're like, oh we got this one cool thing, you do it and then Brexit, we just undid it. Right. It's just all fit.

People would rightly be like, that doesn't seem right. Okay, so we've done the bunking and we've done the debunking and now we have to do a little bit of rebunking. What's getting rebunked? So okay, so that Guardian article that you read that was like, deworming has been completely debunked and like it's it's gone, right? in the sort of worm war spheres for being a little overstatement.

Mm-hmm. Journalists only know how to tell two stories about international development. It's either this is going to solve everything or this is a debunked fucking lie. Mm-hmm. We went through this with microcredit, remember? It was like cover of Newsweek, microcredit's gonna solve poverty.

And then it like they expanded it again too quickly. They expanded it to areas where it didn't work as well. And then it was like microcredits a fucking scam. Yeah. It's like, No, it neither one of those are true. Neither one of those are true. And also like

You only get those two stories if you keep buying into the first one that's like, this is gonna fix everything. Exactly. And also, I mean, my takeaway from this original 2004 paper that has been dissected so much is that like after all the sort of statistical reanalysis. The schools that got the deworming treatments have a roughly 6% higher attendance rate than the schools that didn't get deworming. To me, it's like that is a promising result. It's not bullshit.

And it's not gonna save the world. It's promising. It feels like a grounded result. Right. If you think about your life and sort of things that have made differences in your life, my guess is that it's a constellation of a lot of six percent things. Exactly. It's the kind of study to me that it's like, okay, it

It seemed to quote unquote work in seventy-five schools. Let's try it in another seventy-five schools. Yeah. It seemed to be fine in this region of Kenya. Let's try it next door in Uganda. Yeah. It never should have been expanded to 140 million kids. The the thing that I I wish that I could impress upon people in development aid is that literally nothing is going to be so good and so proven that you can go from seventy five schools to fucking three million schools on another continent.

Yeah. You're never gonna find it. I'll say in public policy advocacy world, which is the world that I come from. There is such a deep scarcity mindset. Yeah. Right. That you start from a place of constraints rather than starting from a place of like Okay, if we really wanted to get to the bottom of school attendance, what would that take? Instead, we start from a place of like, we found this one thing and that does it and we're just gonna get a shit ton of it out there at the end.

And just hope for the best, right? Right. That we get to the passage or non passage of a policy and we think that's the end of the story and actually that's exactly when the rubber hits the road. Right. Like you have to pay just as much attention to implementation, the least exciting sounding thing. I know. But like that's where the promise of the policy comes to life, right? Yeah. And in this one, it feels like

folks just completely skipped over program evaluation and assessment stuff. Right. Well, I mean, there are places where this works. There's a a big study in Uganda that finds that it does increase kids' heights and weights. And it does increase their test scores. Although it doesn't affect their school attendance, which I thought was really funny. So it's like it's like the mirror image of the Kenya study. But it's like, okay, so it it works on these metrics in this place.

The the by far the most interesting study I read on this was there's a meta-analysis of a hundred and fifty-three deworming projects in Kenya. Uh-huh. And the whole paper is about like, okay, what makes this work? Because every development intervention works in some places and not others. And it's not always obvious where it's going to work. Like you said, implementation, this is the hard.

And what they find in Kenya is that the the specifics of the implementation really matter. So basically every deworming project that only gives kids one dose. of deworming pills doesn't work because the deworming pills, they kill all the worms that are inside of you, just like napalms, your entire digestive system. But it doesn't prevent you from getting reinfected.

So if you only go there and give kids one round of pills and you come back in six months and they all have worms again. Yeah. And it really only works in areas where there's like really high infection levels because again If you have like a light or moderate worm infection, like it sounds really gross to think about, but like it doesn't it often doesn't cause any symptoms. Like the worm just like lives in your body for a year and then it dies and you poop it out and that's it.

So unless you're dealing with an area where there's like large numbers of heavy infections, like in the original study, it was like a 22% heavy infection rate. Like that's exactly the kind of place where you should be doing deworming. Like that's that's really debilitating for kids. Yeah. But if it's in a place where like, you know, ten percent of people have worm infections, only maybe one percent of people have heavy worm infections.

You don't want to give pills to every single kid for that, right? It's like giving every single kid a Tylenol for one kid having a headache. Yeah. Another thing they find in Kenya is that it works much better when it's coupled with other interventions.

So when you're giving kids like these deworming pills, even if they do have heavy infections, the worms are gone. But then they need extra food to kind of get like ketchup growth because oftentimes they've had like stunted growth and stunted weight. So it's like they need to eat more for like a couple of months after that to like catch up to where they should be.

But if these are really poor kids and they're in a really poor area and they don't have access to food. Right. Good luck. Yeah, like it it's kinda like there's no point in deworming them because whatever benefits they would get from deworming, they they won't actually get'cause there's not enough food around for them to eat. So it's like you also have to do these nutritional interventions. And in general, deworming works a lot better when it's coupled with

this like much more holistic program about like sanitation, like getting people running water. Also some education programs are part of this. I'm skeptical of those in general, but apparently they work here in some circumstances. So it's like deworming as part of a larger

program of like we're gonna work on improved sanitation in this area, then like, yeah, they're a really essential component of that because you need to get the fucking worms out of the kids. Yeah. But then you're back to an expensive project. And you're back to a project where like you have to actually know the local area and you have to talk to people beforehand and you have to build relationships and donors don't like that. And like the the TED Talk Industrial Complex Davos

They don't like that either. I mean, I think part of the reason that that stuff feels so frustrating to folks is that it feels so slow. compared to I got a million pills out. Right. What I would argue is that building something slowly and right and making sure that it works. actually probably takes less time than going on five of these wild goose chase scenarios. Geez, of course. Everybody thinks we found the silver bullet for whatever.

It's the like public policy and international aid equivalent of a get rich quick scheme. Oh it is. The the most chilling thing to me about this is the way that

This generation of development aid has replicated exactly the mistakes of the previous generation. Woo, say more about that. So when Michael Kramer got into development during the statistical dark ages, right? Everything was kind of like gut-based, vibe-based, right? And they've now created a situation where they're basically saying, like, we're just gonna ignore all of the data. The World Health Organization is still doing mass deworming.

The World Bank still stands behind this, the Copenhagen consensus, the Givewell charity, which is like a group that like I really respect. has written these weird defenses of deworming where they're like, no, no, no, no, no, ignore all the meta-analyses.

What we're really interested in is like the 10 year follow-up study. We know that deworming works because of the 10 year follow-up study, but it's now it's like there's no there's no plausible path to them having better lives. Like it's it it it's completely deranged.

To say that a program of deworming in which like up to half the kids didn't even participate, like didn't even get the deworming pills, and had no effect on health, no effect on test scores, and an extremely modest effect on attendance. Would be affecting people's earnings 10 years later. I'm sorry. Yeah. It isn't plausible to me. And also, it's very weird for this entire field of

development, the whole thing was like the randomistas. That's what they called this like this generation of mostly economists who came into development aid and were like, let's do this in a more rigorous way. That now these same randomistas, these like rigorous people, are now telling me to ignore meta-analyses on the basis of a single study? Like that is so chilling to me. It's like, what was the point of all this? Yeah. I mean imagine being like

Definitely don't pay attention to a fucking cochrun review. Right. This is okay, this is this is what Michael Kramer like during the whole kind of worm war, everybody fighting on the internet period in twenty fifteen. This isn't a sort of response to the response to the response to the response. Like this whole thing is like

fucking Tumblr comments in like twenty fourteen. He's talking about like defending deworming. He says the estimated value of benefits exceeds the cost by more than a hundredfold. While the results of any one study should not be taken in isolation and effects may differ across environments Even if policymakers believe there's a one in one hundred chance of experiencing effects of this magnitude, the expected benefits of deworming would exceed the cost.

They're playing our song, Mike. He's basically saying, like, even if this doesn't work, look how cheap it is. Look how cost effective this is. This is like the asbestos argument. Give well says the same thing. Give well says, because mass deworming is so cheap, there's a good case for donating to support deworming even when in substantial doubt about the evidence. This is very similar to some of the um responses to the calorie labeling stuff. Oh yeah? Yeah, absolutely. That people are like

Well, it doesn't work for everybody, but look how cheap and effective. Right. And you're like effective measured by what? Right. Effective how. We're doing something, we're not doing nothing. But also the something that you're doing has the effect of nothing. Right. And also plus maybe negative. Like what? I also uh I forgot to mention uh Michael Kramer won the Nobel Prize in 2019. Come on, Michael! It's not just for this, it's for like other work and it's for like this whole kind of

the the systematization of statistics within international development. And he shared it with some other economists and like I actually think that like using more randomized controlled trials in development is a good thing. I think statistical rigor is a good thing. But also like that has come with drawbacks and side effects and a lot of like weird and bad cul deception.

I also think to sort of like defend the original 2004 paper, there's not going to be a perfect methodology ever. Sure. Like you can find things to quibble with. in anything, in like a meta-analysis, in any individual study. And ultimately, every single thing that we do in public health is going to be a judgment call. In principle, that's not necessarily a bad thing. I think the idea of a judgment free science.

is always going to be a myth. You're always going to have to have values. Yeah, and if you don't get explicit about articulating and committing to your own set of values. Then values will show up in your work. Yeah, exactly. Yeah. That's a good way to put it. Without your intending them to, right? So if we're thinking about, For example, pretty much all of the like weight loss research that we have out in the world.

starts from a place of assuming that fat people's experiences are immaterial. Right. If I asked some of those researchers like, Hey, do you think that like the life experiences of people who are trying the things that you're testing out matter? I imagine quite a few of them would be like, well yes. Right. But they haven't designed a thing that reflects that. Right. And like that's part of what we're getting into here is

This is research that was not designed in a way that reflected an understanding that there was knowledge on the ground that they needed to know about. Right. There are a bunch of things that again, if you don't articulate those values and then take the next step of going And here's how we're gonna operationalize them. Right. Other values will speak on your behalf. I think the parallel to fat people is really good because

It's driven by the same thing of like this is for their own good and like I'm helping. It's driven by paternalism. Yeah. And I I I think people give themselves permission to like not think things through. To not think things through and to not ask for feedback. Right. That's the like toxic good guy narrative that worries me so much in international development and in public health and in lots and lots and lots of places. So that was uh that was deworming. Is that where we're gonna end it?

Just me being like Everything's fucked and we're all done and we all lost it and you're like, uh, there we go. I feel like okay. So you this is kind of a digression. You posted on the maintenance phase Instagram the other day. It was like a meme about how every millennial podcast is just one person talking excitedly and the other person going, That's fucked up every five minutes. I was like, wow, they've really got our number. I think we both have the same reaction of like owned.

Fair enough. When you're right, you're right. I was trying I was trying to structure something around like how can I get Aubrey to say like that's fucked up. It's fucked up. It's a millennial podcast. That's fucked up. We did it. We got there.