Welcome to the Improving Development Evaluation Podcast. I'm your host David Wand, and welcome to episode eight, part two, where we continue our discussion of the international development organization called United Nations Development Program, or UNDP.
You can learn more about the United Nations Development Program at UNDP.org, and we're looking at government of Canada funded project cost to the taxpayer of $30 million in Bangladesh, entitled Fighting Environmental Degradation and Promoting Greener Energy. And in summary, basically going to the Cox's Bazaar refugee camp, but also communities outside of the camp, giving them liquefied petroleum gas to replace their need to go into the sacred bush and get firewood.
So basically, that's the idea behind the project. And as I will elaborate further, there are lots of expected outcomes from replacing firewood with LPG. One of them is improved nutritional status for the children because you can cook more frequently nutritious meals. And also, it's a safety issue for women.
They would not have to go out into the forest and collect firewood, which reduces the risk or eliminates it if they don't have to go out at all because they have LPG at home, reduces the risk of sexual assault and rape. And the other thing it does is it reduces the harmful smoke from burning firewood. So those are some of the expected outcomes of replacing the need to go out and get firewood when you have it replaced by LPG.
But another expected outcome I'll talk about in more detail is you hope, since we're worried about the environment, that in the longer term, as I mentioned in part one, they'll replace the LPG with solar cookers or biogas.
And although that conflicts with perhaps Canadian LPG companies, because they're gonna lose business if that happens, but ideally that's what you'd like to see, even though the Canadian LPG companies, assuming they're using Canadian LPG companies to supply this camp, they're gonna lose a customer. But in the long run, that's gonna benefit the environment. So let's get with it.
So some background is on June 10th, I sent the performance measurement framework and the critique of the performance measurement framework, which basically said that of the nine outcome indicators, measuring the three outcomes in the performance measurement framework, I only found two out of the nine that were properly measuring the outcomes. And on June 10th, as I said, I sent it to three departments of the UNDP.
I sent it to the Independent Evaluation Office in New York, on June 10th, also sent it to the United Nations Development Program in Bangladesh, their country office, because that's where the project is being implemented. And I also sent it to the United Nations Development Program, Asia and Pacific Regional Communications Advisor. On June 18th, I got one email reply from the Asia and Pacific Regional Communications Advisor, and he advised me, he was on vacation, so he couldn't respond.
He told me to email the Bureau for External Relations and Advocacy, UNDP in New York, which I did, and he confirmed the right email address. I sent that also on June 18th, but as of today, June 28th, I've received no reply from any of those three UNDP departments. And I have also not received any written response to my critique, even though in the email on June 10th, I said that was another option.
If they didn't want to be on the podcast, this part two, to discuss my critique, they could send a written response, but I haven't received that. So on June 28th, which is today, I'm now going to do part two, unfortunately, without them, and without a written response.
And then at the end of that, I will, as I usually do, send it to the Minister for International Development, as well as the shadow critics from the Conservative Party, the NDP, and the Bloc Québécois, and also the Green Party, so everybody knows, because this is not a political particular affiliation that you need to have. Everyone of every political persuasion should be concerned about the fact that Canadian government funds are not being properly evaluated. So let's get on with it.
So there are nine outcome indicators measuring the three outcomes. So we'll start with the first outcome, number one, and I quote from the performance measurement framework. Targeted households have safe and sufficient access to cleaner cooking energy to meet their basic needs. Problem there is, that's an output. You're just delivering the LPG. It's not an outcome. It's after you deliver the gas, what are the expected outcomes? And I've already mentioned some possible expected outcomes.
So that needs to be changed and reworded. And it could, to some effect, be, like I've described, forest areas where refugees harvest firewood is reduced and replaced by seedlings. That could be an example of an expected outcome. But there's other expected outcomes I mentioned earlier. But let's get to the outcome indicator for this flawed outcome, number one. And that outcome indicator, number one, is percent of households who report buying or collecting firewood.
The problem there is, it's all full of self-reporting bias, but even more importantly, it is forget about it. Don't even measure the indicator. Don't waste your time asking households how much firewood they bought or collected. Go right to the chase and look at some of the expected outcomes instead. Weigh the children to see if their nutritional status has improved.
Ask the women how many times they've had to go out and collect firewood, because you expect that to go down, if they've got the LPG in their house, in their location in the refugee camp, right? So those are better indicators rather than just asking them if they've been collecting firewood. And the other one is, look at the children or the adults to see if they have the LPG, if smoke inhalation-related illnesses related to burning firewood have gone down.
So those would be the basic things you would look at instead of running around trying to figure out how much firewood they've got in their location or how much they're collecting. So it's not a very good indicator, and it creates a lot of waste in terms of measurement. So you could just do what everybody else is doing, which is weighing the children anyways, right? So that's where I recommend instead of that indicator, focusing on those as alternate indicators.
The second indicator, this is one of the two that's really good, and they've mentioned it, the number of equivalent hectares of natural forest protected. That's an excellent indicator. So I've noted that in the critique, and they actually have a target of 10,700 hectares of natural forest protected. And that's actually in the performance measurement framework. So that's excellent. They even indicate in the performance measurement framework that they will be planting seedlings.
So the key there is to make sure that that target is reached, and I'll talk about that a little bit later. So that's an excellent indicator. Forget about the first one. I mean, that's just a waste of money. And leading on to that, the third indicator is the number of tons of carbon dioxide emissions abated. So good luck with that. I would again say forget about it. Don't even waste Canadian taxpayer money measuring that.
Focus on the forest that they're going to to collect firewood and monitor it by making sure that seedlings are replacing where they in the past at the baseline ripped out the wood, chopping down the tree. The other point is how are you gonna claim that any reduction in carbon dioxide, because it's floating in the air, I don't know how they do this, how do they claim that it's been abated?
The only way to show that, and maybe that's what they're implying here, is by the forest where they used to harvest the firewood is now being reduced, and they can monitor that area around the camp to show that. So that's what they should be doing. So take out that indicator. And in fact, in outcome number two, this is what they do, and I'll read the outcome. Number two, environment and ecosystems rehabilitated. Now that's the good part. The next part is a bit flaky.
To strengthen community resilience to mitigate conflict over natural resources, disasters, and climate shocks. The first indicator for that outcome is, and this is indicator number four, is the percent of land ecosystems rehabilitated in Cox's Bazaar through SAFE Plus Two program interventions. SAFE is the name of the NGO locally that has been hired by UNDP to bring in the liquefied petroleum gas. So that's an excellent indicator.
And in fact, if you look at the performance measurement framework, they actually have, I think it's called geographic information system monitoring. They use satellite imagery of that specific area of the forest where the refugees used to go to get the firewood, and they're planting seedlings in there to replace the firewood that was ripped out. So they just need to track it and to show that that target of 10,700 hectares is being protected and sustained. And that's excellent.
So that's a great indicator. That's the second indicator, which is pretty good. And the first one, which is equivalent basically to it, is the number of equivalent hectares natural forest protected. It's basically the same thing. So it's great that they're doing that. So that's an excellent indicator. Indicator number five, again, is a problem. It's tons of carbon dioxide sequestered through program interventions.
Again, how are you gonna claim that the LPG coming in has actually sequestered the carbon? Now their response could be, duh, it is the seedlings that we're planting. Okay, I get it. But actually measuring the carbon dioxide sequestered, waste of time and waste of money. Just focus on the forest around the refugee camp, which they have done, and show that it's been sustained. But also, as I mentioned earlier, look at the other expected outcomes from bringing in the LPG.
And then this is where it gets really a lot of waste and fun and time and money. The next indicator they've got on the environment and ecosystems rehabilitated to, quote, strengthen community resilience to mitigate conflict over natural resources, disasters, and climate shocks. The next indicator they have is, quote, resilience capacity score increased compared to baseline.
The resilience capacity score, it measures households' perception of their resilience capacities to generic or country-specific shocks and stressors. And this again is where I say forget about it. Don't ask about their perception on fighting over firewood. Don't do it. Just look at, instead of asking for their perception to become resilient, which is a waste of time and money in measuring it, it's also got self-reporting bias.
Weigh the children, because you're expecting their nutritional status to go up. Inspect the children, for lack of a better word, to see if the smoke inhalation-related illnesses related to burning firewood has gone down. So all the symptoms of that have been reduced. Also ask the women if they've had to go out and get firewood, because if they, of course they could lie, but what you wanna look at is the frequency of reporting going out and collecting firewood has gone down.
Those are better measures, direct objective measures to see that the LPG coming in is actually having the desired outcome, right? Forget about this resilience capacity score. You don't need it. So there's another outcome number three, which is the resilience of Rohingya refugees, especially women, is sustained. And there they've got an indicator, which is the resilience capacity score is increased compared to the baseline.
Again, you could ask the women what their perception is of fear of going out and collecting firewood, but even better, just get to the frequency that they're going out and collecting firewood, because hopefully it's gone down. You could even go a little step further, but I wouldn't even try it, which is looking at the number of reported, because that's tricky, reported incidents of sexual assault and rape that have occurred in the area where the firewood is being collected, right?
It's better to focus on the outcomes I described earlier. And another indicator they've got, this is indicator number eight, is the average reduced coping strategy index. The coping strategies index is a tool that measures what people do when they cannot access enough food. It is a series of questions about how households manage to cope with a shortfall in food for consumption and results in a simple numeric score. Now the problem here again is you have to look at the output.
Now in addition to bringing in liquefied petroleum gas, taxpayers from Canada, $30 million, also are spending money in this project on cash. That is giving cash to buy food in the refugee camp and also cash, I mean from the $30 million, to train the refugees in growing their own food in a community garden. And they only measure this average reduced coping strategy index once a year. That's not good enough, that's the first problem.
The second problem is, hello, if you're giving them cash to buy food, don't worry about this coping strategy index, figure out if they're buying the food with it and you can do that at the camp. I happen to be at the camp. I've seen the World Food Program in action. The refugees have a digital card where they go into stores inside the camp and buy food and it's registered on these cards. So the World Food Program has detailed data on the amount of food that's actually been bought.
And again, like I said earlier, they can also weigh the children in these families to see if their nutritional status is going up. There's no need to have this average reduced coping strategy index measured, especially just once a year. It's a waste of time and a waste of money because the idea here is to increase their nutritional status. So you do it every six months, I would recommend.
Forget about this strategy index and measure the amount of food they've actually bought and or weigh the children. And you can even weigh the adults to show this is standard nutritional status measures. You don't have to have this funky, flaky coping strategy index. Also measure the garden crop yields if you want on the training on how to grow crops. But I would even argue who cares about that. Go to the end expected outcome, which is increased weight gain, nutritional status of the children.
And also the cash purchases of the food items. I'm repeating myself here, but that's the idea. So no need for this coping strategies index, especially when the key outputs in this project are LPG and cash for food and training to grow food. So just measure the crop yields, food consumption, and the weights of the children. And that would be good enough. The final indicator on this outcome of the resilience of Rohingya refugees, especially women is sustained.
Final outcome indicator is percent of households with acceptable food consumption score. Now again, it's only measured once per year. And I would say you need to measure it more frequently. And it's a consumption score. Maybe they use the data from the World Food Program to be fair to them, but we don't know because they haven't given me a written response or showed up for this part too. So maybe they measure acceptable food consumption with the data that comes from the World Food Program.
I'm not, I just don't know. It's only measured once per year. I would rather go in and measure every six months the weights of the children. And that would show their nutritional status going up. So that's the end of episode eight, part two. What I'm gonna do now is send this to the minister and the shadow critics as well as UNDP. I will copy them on that email so they're aware.
And if you are interested in receiving the performance measurement framework and the critique that I've verbally just described here, you can send me an email at evaluatecanadaaidatgmail.com. And I'd be happy to send it to you. I can also send you any of the other performance measurement frameworks for season one. This ends season one for the podcast. I will be returning in September for season two.
And before I do that, I will be doing one more episode, sort of a prologue of season one, which gives you an idea of how season two is going to be slightly different than season one and give you some hints as to what I'm expanding on and maybe what I would be refocusing on. And I'll probably launch that in a few weeks. Thank you for listening and thank you for the feedback.
I've received some very positive emails regarding potential learning opportunities on the podcast, which I'm thinking about, but I'll elaborate more on that in the prologue to season one. So that's it. Thank you for listening and stay tuned for the prologue for season one and also season two, which will start up again in September. Bye for now.