Welcome to the Improving Development Evaluation Podcast. I'm your host David Wand and welcome to this episode where we feature World Vision Canada. You can learn more about World Vision Canada at their website worldvisioncanada .ca. This project we're going to talk about cost the Canadian taxpayer 41 million dollars and it is delivering services and products in Kenya, Somalia, Bangladesh, Tanzania, and Cambodia, as well as Canada as a country. And it is delivering a variety
of services and products. The title of the project is Realizing Gender Equality, Attitudinal Change, and Transformative Systems in Nutrition. But before I get into the details of the project, I want to give you a little bit of background as to why it's taken me so long to do another
episode. My episodes depend on receiving performance measurement frameworks, PMFs, that list the outcomes that they are claiming they're going to achieve with the project, as well as, more importantly, the outcome indicators, because we need to know if they're valid or not. As I've said before on this podcast, before the money can be dispersed, the $41 million, they have to produce this performance measurement framework to the government of Canada, and the government of Canada has to approve it.
If you go to the Project Browser website, which I'll provide in the episode notes where the project is, you'll notice they've been receiving this $41 million since May of 2023. So the PMF has already been done since 2023. I requested this PMF in October of 2024, but now it's January of 2026 when I finally received the PMF, even though I'd been complaining for several months, even sent a complaint to the Independent Information
Commissioner. It wasn't until I sent an email to my local member of parliament who happens to be a liberal in the government, happens to be the minister of immigration. And I said to her, look, I'm not getting this PMF. Somebody's stalling. Come on. It's been over almost two years since I've received this PMF. What's going on? And I kind of jokingly said, I'm not going to vote for you the next time. Within a week, I received an email from, quote, her team. saying
her team would get back to me. And they did. Next, about a week later, in January of 2026, after initially requesting the PMF in October of 2024, I finally received it. So that's some background for you about the politics of information. So I'll shut up now and get to the details. To give you a bit of background about the target groups and the products and services that this project spent $41 million delivering in these
various developing countries. One of the target groups was Canada and the developing countries I mentioned, their populations. And one of the products they produced was art to raise awareness on gender equality for those populations. Another target group was World Vision staff themselves. They were trained on how to develop message products to raise awareness on gender equality. Probably the World Vision offices in the developing countries, the staff that work in the developing countries.
Another product that the World Vision staff produced was case studies that were completed and disseminated to... gender transforming nutrition, including budgeting and monitoring, and also women's rights groups in these developing countries, another target group. They were trained on how to advocate for nutrition and sexual reproductive health services. Another target group were health officials and health facility managers in these developing
countries. They were trained to supervise health workers on proper gender responsive services, budgeting and monitoring. Health systems was another target group. I don't know what human beings are there, but reports were produced on gaps in health services. Ministry of Health staff were also a target group in these developing countries that were trained on how to plan for anemia prevention and treatment and malnutrition
and family planning. Teachers were another target group in these developing countries, training on adolescent health. Health facilities also received equipment and supplies for nutrition and to deliver sexual reproductive health services, as well as equipment and supplies to deliver water and sanitation. Farmers were another target group in these developing countries. They were given seeds as part of this $41 million to grow
food. Faith leaders in these developing countries were also another target group that were trained on how to reduce barriers for women to access nutrition and sexual reproductive health services. Men and women were another target group that were also trained on how to reduce barriers for women to access nutrition and sexual health services. Boys and girls were another. Target group. They were trained on gender equality. And finally, there were peer networks in these developing
countries. They were trained on how to deliver food and hygiene demonstrations, as well as how to develop messages specific to sexual reproductive health services. So that gives you an idea of what they spent the $41 million on. Now, lots of training, seeds, all sorts of things, equipment, supplies, as I've just described. Now the question is, if we look at their performance measurement
framework, they have some outcomes. In fact, they have outcome statements, 13 outcomes that they expect to achieve after spending that $41 million. And out of those 13 outcomes they expect to achieve, In the performance measurement framework, they developed 38 outcome indicators. So we're going to go through not all of them, but as many of them as we can in the next 30 or 40 minutes.
And then I'll give you an idea of what I came up to as to whether or not those outcome indicators are garbage and they can't support any of the claims that they've achieved those outcomes because the indicators don't even measure what they're trying to claim they're achieving. maybe there's some good outcome indicators in there. It's a
bit of both, as you'll find out. But before I do that, and you live in the Global South, or you hopefully are planning to visit the Global South to hopefully properly evaluate this project, or for that matter, any project, you may want to check out some sunglasses at akila .la, A -K -I -L -A dot L -A. In the episode notes, I'll
give you that website. If you buy them, prescription lenses included in those sunglasses, all made from 50 % recycled material, you will get a 10 % discount if you use my code, WANDCOOLT -T -SHIRTS, or you can just click on the link that I have in the episode notes, and you'll get a 10 % discount. And when you do that, I get a 15 % commission. when you purchase any product using my promo
code. So you have your sunglasses to protect you from the sun and the global south, but you need the right clothes to be comfortable in that hot and humid climate. So what do you want to do there? You want to check out hypernaturalstyle .com. Some men's polos, tees, and button -ups. You get 20 % off any product when you use my promo code WANDCOOLT -SHIRTS. And I get a 20 % commission of the total purchase you make when you use my promo code WANDCOOLT -SHIRTS. So what
is so cool about these t -shirts? Well, they have a patent -pending HyperCool Jade technology. that when it's hot, it lowers your body temperature by three to five degrees. They are also breathable thanks to Supina cotton, and they are sold in premium retailers like Nordstrom, and were voted the best men's polo shirts in 2024 in Men's Journal, Forbes, and Esquire. So let's get on with the outcomes and the outcome indicators. So let's
start with this first. Outcome statement increased engagement and awareness of the Canadian public on the impact of official development assistance in the context of gender equitable and responsive nutrition, health and sexual reproductive health programming. They have two indicators for there. The first one is number of Canadians aged 18 to 65 years of age who are engaging with content related. to nutrition, health, sexual reproductive health, gender issues in international development
contexts. So what's interesting here is that sounds like a pretty good indicator, but then you have to look at the PMF to see what exactly they do to measure it. Well, bad news, dudes. They do not use a survey of the Canadian public. So they can't even claim. that they're achieving their target group, which is raised awareness of the Canadian public. All they talk about is the number of likes and comments online and the number of in -person conversations at events.
And they do it at baseline, midline, headline, and annually. And it's going to show an increase in engagement and awareness. But they do it from the project database. But that's not a sample survey of the, quote, Canadian public. So they can't make the claim. that the Canadian public has experienced an increase in awareness specific to gender equity nutrition, sexual reproductive health, based on Canadian foreign aid has increased. They can't. They've got the wrong target group.
They can claim that the project people that engage with the project, but they haven't done a sample of the general Canadian public population. So they've got the wrong outcome. So they can't make that claim. They could change it. They could reduce it down and just say, You know, if people increase over time their engagement with us, whoever shows up to these events or send comments in online, fine, we could show there's been an
increase in engagement. But they can't make the claim that the Canadian public at large has increased their awareness. Right? And it's the same for the specific to the awareness of official development assistance impact has increased. They've got the wrong target group. Because they haven't done a sample survey, right? The good news is, as you'll find out later, some of the other indicators they've got, they do a very good job because
they actually do do a household survey. So that's an example of two indicators that really are not doing it properly. So they're not good. So if we keep going, we can look at another outcome.
expected to achieve increased utilization of gender transformative framework for nutrition by key national and subnational stakeholders for mainstreaming of gender equality into the planning budgeting monitoring and learning processes of multi -sectoral nutrition programs and policies they've got a few indicators for that you'll see well you can't see you can only hear First one is percent of national, subnational government, civil society, NGO stakeholders endorsing GTFN,
gender transformative framework network, mainstreaming within multisectoral programs and policies. So that's one indicator they have. Another indicator they have is this framework, gender transformative framework nutrition used at least in one subnational level government sector. in the project site for mainstreaming of gender equality into the planning, budgeting, monitoring, and learning processes of multisectoral nutrition program.
But if you can, if you look at their indicators, the first one, they talk about endorsement, endorsing mainstreaming of GTFN. That's not the same as increased utilization. They could say, oh yeah, we've endorsed it, but you've got to look to see if they've actually used it. So it's not a valid indicator. It doesn't validly measure whether actual utilization is increased. If they want to go back and change the outcome statement to endorsed, fine, but they didn't. So they can't
make that claim. The other problem is with the second indicator used in at least one subnational level government sector. If you look at the data source, this is where the PMF is important because you wouldn't know this unless you got the PMF. They look at the project database regularly to see if it's been used, right? Okay. You have to just trust that whoever's entered the information in the database, they've just said, yeah, they've used it. But it's not really a good measure.
I think it would be better that they actually go in to the government sector to see if they've actually used it. You know, developing some criteria. Did they use it or not? They shouldn't be going into a secondary data source as the project database. So I don't trust it. It's a bias, right? You should actually just show up a third party evaluator and pick a government sector, go in and say, did you use the GTFN? Did you use it? And they could check by seeing. actual documents rather
than just going into the database. So that's another two indicators that are not well designed. They can't support the claim that utilization has been increased. Another outcome, increased capacity of local community structures such as citizen voice and action groups, community health committees to engage in evidence -based advocacy and policy dialogue at national and sub -national levels on gender equitable rights. to nutrition, health, and sexual reproductive health rights.
So they've got three indicators here. First one is percent of community -based structures that are monitoring gender -responsive and adolescent -friendly service delivery at health facilities disaggregated by country. Again, if you read that, monitoring, they're equating their ability to monitor with their increased capacity to advocate. That's wrong. They're not the same thing. Now, monitoring is, you could argue, is an indirect measure that they're actually capable of advocating.
You could argue that. So I've given it a green, meaning I say it's okay. They do an assessment of the health staff and look at the project database. We can assume that this assessment looks at the ability to monitor. So it's an indirect measure. I've given it a green light. I would say that's a good indicator for that. outcome of increased capacity. The second one indicator is percent of trained members who understand basic health rights and how these rights are articulated under
local law disaggregated by country. And if you take a look there, they actually have an objective testing measure of this understanding on basic health rights and how to articulate these rights.
So I checked for the existence of an actual test there is a test so that's excellent this is a good this is a green light good indicator again the only issue there is and i again i haven't said it's bad enough to give it a red light is would the increase in that percentage who passed the test be significantly greater than a group of other individuals who didn't participate in the project meaning what i call impact Some people disagree with that definition, but that's the
one I use for the podcast. Not only has it been achieved, effective, but it's been achieved so greatly that the increase on the indicator is so great, it is statistically significant, and someone can argue it's not due to chance. But we're not looking at that right now. But that's a good indicator. So the last indicator on that outcome is extent of engagement of women and
men. community leaders in targeted villages with local government on nutrition, health, sexual reproductive health rights, and gender issues. Now, for that one, if you look again at the PMF, if you don't have the PMF, you can't find this. But then you look at the data source, and this is where it gets, tell me about your feelings. Engagement is measured with the community leaders themselves. So there's a bias. And using focus group discussions. So it's going to lead to self
-reporting bias. So for this indicator, I gave them a red light. Not a good measure. Focus groups. They need to get a better measure of engagement externally to show that the level of engagement is actually quantitatively increased. So that's not good. So not bad. Three indicators. Two of them really good. One of them bad. So if we look
at another outcome, improved. capacity of sub -national health management teams for effective gender equitable and responsive nutrition health and sexual reproductive health systems governance including planning budgeting reporting so improved capacity again they have three indicators the first one is called percent of health facility scoring greater than 80 on their health facility planning assessment by the team This indicator
is good. We can only assume that the test validly measures the technical capacity of the team to do, quote, planning, budgeting, and reporting. But again, I've given it a green light, but also the impact issue is, could the team, without this $40 million, been able to teach themselves how to do this planning, budgeting, and reporting? So that should be put in the PMF. is how are they going to show that that increase in percent
is statistically significant, right? Now, it could be as simple as just measuring repeatedly over time, showing the percent going up, and that's good enough. You don't need a control group, but there should be some sort of quantitative analysis to show that it's been going up, and not only going up, scoring greater than 80. but statistically significantly going up. So that's another way to look at it. But I gave them a green light for that indicator. The next two
indicators, I gave red lights. So the next indicator is percent of health workers who report they trust the leadership of the management teams in ensuring effective gender equitable services. Some people will say, oh, we're doing that to
see if we can check against. improved capacity of those health management teams i gave it a red light because it's if you look at the data source it's the health team members themselves so there's self -reporting bias there and also measuring trust has nothing to do with increased technical capacity to deliver responsive budgeting etc right so that that's not a good indicator for those two reasons self -reporting bias because they're asking themselves If they trust themselves,
you have to look at the data source, right? When they say health workers, according to the data source, it's not health workers. It's the health team members themselves. But even if I'm wrong there, I wouldn't use it. I'd throw it out. The third indicator for this outcome was percent of team members who have access to planning, budgeting, and governance data. And here we go again. who report by themselves using the data to make decisions about the provision of gender
equitable services. So instead, we've got self -reporting bias again. What I would do instead, this is not a good indicator, is I would actually sample the reports to see if the data was actually used to, quote, make decisions of gender equitable services, right? I'd do that by an external evaluator instead. Self -reporting bias. So out of those three indicators, only one was good. The other
two were bad. Another outcome, increased health, infrastructure, information technology, and capacity for prevention and management of acute malnutrition in most marginalized communities. The problem here is three indicators. Number of school -going adolescent girls who received the recommended scheme of weekly iron folic acid supplementation. Second indicator, percent of children admitted treated for moderate or severe acute malnutrition.
And third indicator was percent of health facilities in remote catchment areas rehabilitated received support for HMIS strengthening to improve gender equitable nutrition. Now, they're claiming here that if you just deliver those services to the girls and the children, that that in turn can be equated with increased capacity for prevention and management. You could argue that. I'm changing my mind here. I'm thinking these are just outputs.
But what they're trying to tell you here is that if we deliver that folic acid and we deliver that treatment to the children for malnutrition, and if we deliver to those health facilities support, whatever that means, then we could argue... that that in turn is the same as increased capacity to prevent and manage. So I'm going to change my mind here and give this a green light instead of a red light, because it is a good way of saying we're delivering these services and those services,
just getting them delivered. It is charity, but that's okay. That's another issue. So I'll leave it at that as a valid measure of achieving that outcome. Another one is increased knowledge and skills of health care providers on gender equitable and responsive nutrition, health and sexual reproductive health services. Now, again, this is the FGD problem. The first they've got, how many indicators have they got for this outcome? They've got three.
So the first one, extent of change in knowledge and skills among health care providers after receiving training. Fantastic. Problem is, go to the data source. They're not measuring them objectively on their knowledge and skill levels. They're holding a focus group discussion. Instead, they're holding key informant interviews. Instead, no need to do it. So it's not important. Only use focus group discussions, key informant interviews, if your quantitative indicator is not achieving
your outcome. Then you can go back to the group and say, why? What's going on here? What's the problem? So if you look at the second indicator, it's excellent. Percent of health workers who have passed knowledge competency tests following training. It's great. It's a pre and post test. So that's very good. Love it. Fantastic. Third one indicator for increased knowledge and skills is percent of households received at least one visit. from a community health worker in the
last month. Well, they could show up and, like me, when I was in Sierra Leone, have a jug of palm wine and have no knowledge and no skills and get a free meal. Cassava leaf and rice, my favorite dish. Right? So this indicator, in my opinion, is useless. Third indicator, percent of households receiving a visit has nothing to do with increased knowledge of health worker you just trained. This is an output. So we don't know. That health care provider actually has
increased levels of knowledge and skill. Unless one of the measures of knowledge and skill is how to find the health center, which on a motorcycle in Sierra Leone can be a bit of a challenge. Took me about six months to figure that out. But no, in this case, it's not good. They should just use that previous indicator, pre and post
test, good enough. Another outcome, increased gender equitable access to Nutrition -sensitive interventions in wash, water, sanitation, and agriculture, including production and consumption of biofortified crops and nutrient -dense local vegetable and fruits. This one's interesting. The indicator for this in terms of increased access is the percent of mothers planted biofortified crops after receiving planting material directly.
Now, you'd assume... They're going to have access to this food after they plant the seeds, right? But they ask the mothers. Don't ask the mothers. Just say to the mothers, excuse me, show me your plot. So don't survey the mothers. Instead, they should survey the crops to see if they were planted and see if the crops survived, right? So that's a no -go. Now, they do have project progress reports. They might show this, but we do not know. All we can see is the indicator is useless.
Same for the second one. Percent of mothers who planted local vegetables after receiving the seeds. Again, they surveyed the mothers. They shouldn't. They should just survey the vegetables to see if one, they were planted and two, to see if the vegetables survived, right? And the third one, same problem, planted fruit trees. So I'm assuming from the data source, they're asking the mothers instead of going. Now, maybe it's in the survey manual. When you ask the mother,
she says yes. Then go outside and check the plot and to see if the fruit trees were planted and survived. So right from that reading, they've got the wrong data source. Maybe World Vision, if they have the courage to show up, they'll say, David, we can tell you by fact that in the manual, when they did the survey, they went outside and checked. But I don't know. It's funny because
the last indicator, there's another one. percent of households with hygienic toilets after receiving information on appropriate water and sanitation and hygiene. Wash. We can assume that the household survey includes a question. Is there a hygienic toilet? So that one, they can actually go and see it. So they should do the same for the vegetables, for the fruits, and for the fortified crops, right? So out of the four indicators, the hygienic toilet is a good one. The other three are not.
So let's see, what are we running out of time here? How much time have we got here? Well, we're at 30 minutes. We can keep going for a bit. I think I'll stop around 40. Here's another one. Increase knowledge of key community -level influencers to challenge discriminatory social norms that limit women and adolescent girls from accessing gender -equitable, nutrition -specific health, WASH, and sexual reproductive health services. And they've got, I think they got three indicators
for this outcome. And the first one, percent of mothers of children under five who know at least three modern family planning methods after receiving the information from religious leaders and lead fathers. It's a good indicator. It checks on whether that knowledge transfer from the influencer to the mother actually happened. Only issue again is, as I've mentioned before, is could that have happened without spending $41 million? Could they have figured that out, modern family planning
methods, without spending $41 million? By talking to other mothers, going down the street, whatever. So what they should insert in the PMF is, over time, was that percent that increased, was it statistically significant, one -tailed, right -tailed, P less than .05, right? Or they could use a comparison group, but they don't have to, of a group of mothers. But they should be putting that in the PMF to show, yeah, the percent not only went up, but it went up statistically significantly.
Therefore, using my definition, it has, quote, impact. It's not just a difference, effective, but the difference was so big, they could say, wow, it's an impact. I know that DAC uses impact broader. They say unintended and intended effects above and beyond the project in other areas.
That's not my definition. Second indicator, changes in perception and opinions among key community influencers to challenge discriminatory social norms that limit women and adolescent girls from accessing gender -equitable, nutrition -specific health, wash, and sexual reproductive health services. The problem there is the mother's knowledge is first dependent upon the influencer that gives
it to them, also having that knowledge. Where is that test of the influencer's knowledge, like the knowledge test they gave to the health worker previously? This is missing and should be done. Also, qualitative focus group discussions with these influencers is not a valid measure of perception change. Need to include in the test on knowledge. that perception change. You can do that quantitatively.
So it's not a good indicator. The last one, percent of mothers of children under five who know at least three benefits of exclusive breastfeeding after receiving the information from religious leaders and lead fathers. Again, excellent indicator. The only issue again is, is it statistically significant compared to if we didn't spend that $41 million at all? Could that percent go up
anyways? So we've got to make sure if we're going to blow $41 million that our intervention is increasing that percent way, way, way, way better of those mothers that know about exclusive breastfeeding and how it's beneficial. That percent of mothers goes way up higher because of the project rather
than if we didn't have the project at all. So we'll move on to... A few more indicators, improved knowledge and self -efficacy of poor and most marginalized women and adolescent girls to negotiate access to and control over nutrition -specific and sensitive health and sexual reproductive health rights services. We got three indicators there. But again, the indicator, first one, percent of women who decided to use family planning alone or jointly with their partners. Sounds like a
pretty good indicator. But what they do, if you look at the PMF, is they indicate they've got target areas. So that means that there's areas where they're not operating to increase this knowledge and self -efficacy to negotiate access, right? So they need to show that in the non -project areas that that indicator, percent of women who decided to use family planning, is statistically significantly lower than the target areas. It's
got to show impact, right? Because if the project $41 million spent doesn't really make a statistically significant difference in that percent, it brings into question the efficacy and impact of the project. Second indicator, percent of school -going adolescent girls and boys with appropriate knowledge. on adolescent nutrition, sexual reproductive health, et cetera. Now, what have we got here?
I think we don't know. I must confess, I don't have the answer here because I'd have to look in the PMF and that will take a bit of time. So we'll look at the third indicator. Again, the same issue, percent of mothers with appropriate knowledge on good maternal nutrition before, during, and after pregnancy. Again, The indicator is excellent, but the percent is measured annually,
just annually. That suggests for this indicator that throughout that year, before they measure it, the group of mothers that aren't exposed to the project, they also could figure out the knowledge on good maternal nutrition before, during, and after pregnancy. So they have to put in the PMF how their project... They just got to mention control group, comparison group, or statistical significance analysis on the percent over time, right? They just got to put that in
there. As far as I know, it's not in the PMF. So I'm going to assume they're not looking at it and they need to look at it. They want to make the claim that their project increased the percent statistically significantly with impact.
Next outcome. improved effectiveness of local stakeholders in target countries and Canada on gender equitable local and international nutrition specific and sexual reproductive health rights activities advocacy and policy dialogue so the outcome indicators there are Canadians reached through the project on nutrition health sexual reproductive health and gender equality issues so it's a little vague How do they define improved effectiveness? They don't. Effective in doing
what? Increase in knowledge? But its outcome indicator is clear, specific to increasing reach. Canadians reached. The measure of the number reached is clear, based on counting the number online, who have read, watched, commented, shared,
donated through their website. awareness campaign but there's no data on whether their quote awareness campaign in this project was responsible for a significant increase in this number so i gave it a negative there not very good in showing their project increased reach more than if they didn't do it same problem with the second indicator extent of engagement of local health facilities on gender equitable nutrition, health, water, sanitation, health, and sexual reproductive health
rights issues. Data source and data analysis is a focus group discussion, which is a poor way to measure whether levels of engagement of stakeholders with health facilities, specific to if services have been delivered equitably. They need to better define and measure improved
effectiveness. specific to levels of engagement and the last one is my favorite degree of effectiveness of local women and adolescent girls rights groups and associations feel using a liquored five point scale in ensuring gender equitable nutrition health and sexual reproductive health services do not ask them how they feel about their effectiveness in ensuring that services are delivered equitably. Measure their ability to advocate for services
to be delivered equitably. For example, evaluate written advocacy products they have written or their presentation of those advocacy products. That would be a better measure of how effective they are in ensuring gender equitable services are delivered, right? So on that one, those three
indicators get a red light, unfortunately. So if we go to another one, strengthened delivery of gender equitable and responsive nutrition health and sexual reproductive health services for the poorest and the most marginalized women, adolescent girls and children. The indicator, percent of adolescent girls who report that they were offered services without judgment by providers. Excellent, valid indicator. And it's using a
household survey. That's awesome. But again, they have to show, using that household survey, if they used a right -tailed, one -tailed statistical significance test to show that their $40 million project increased that percent statistically and significantly over time compared to where the project was not operating or compared to
the same target group over time. Excellent indicator, but there's no indication in the PMF that they're doing any statistical analysis to show that it's had a statistically significant difference in the right direction, percent increase over time. But it's great. It's expensive to do a household survey. So you would think they're doing that
analysis, and maybe they are. It's just not in the performance measurement framework, but it should be, especially when the performance measurement framework is trying to show If the outcomes have been effective and have been achieved. The second indicator, percent of health facilities promoting gender equitable and responsive nutrition as measured by a modified tool in the baseline. Now, again, without the performance measurement framework, you'd think, ah, that's pretty good.
But then if you look at the PMF and you look at the data source, it's quote, the head of the health facility. So obviously the head's going to say, yeah, it's great. We did it. We promoted gender equitable services. We did it. Yes or no? Of course he or she's going to say yes. Conflict of interest. They should have an objective measure of determining if the health facility is actually promoting gender equitable services by looking for, quote, gender equitable products at the
facility. Or they could take out the syndicator. Since the survey of the patients visiting the facility is covered in the household facility. Household facility survey. If they, the girls, the name of the health facility they visited to get health services. They could actually in the survey of the girls, in the household survey, ask them which health facility did you go to? Could have a whole list in the region and they could just tick it off. And that way they could
show. If there's been an increase in the health facilities, that would be a better measure rather than just asking the head of the health facility. The last one indicator changes in perception of female and male clients on the gender responsiveness of health facilities in providing nutrition, health, wash, and sexual reproductive health rights services. So again, the problem here is if you look at the data source, It's a focus group discussion. It's a poor measure of tracking
changes in perception. Ask any social psychologist. You can easily do that quantitatively. Again, even changes in perception in the desired direction could happen anyway outside of the project. So what they need to do there is do a quantitative and ideally do some stat significance to see if the percent has gone up. in perception in the right direction, right? So how are we doing here? Oh, we're almost done. We're getting to the global indicators now. So here's another
outcome. Improved adoption of gender equitable practices in nutrition, health, and sexual reproductive rights at individual, household, and community levels. And if you look at the indicators here, These are definitely globally recognized. So the challenge is connecting the $40 million project with whether or not that $40 million project achieved these three indicators. The first one, percent of mothers of children who attended at least four antenatal visits during their last
pregnancy. Project outputs clearly deliver services. The only challenge is to show if the change in those percents on the indicators is due to the project to show the significant impact of the
project. Therefore, the PMF needs to show in the data collection method a one -tailed, right -tailed hypothesis where the null hypothesis is that the observed percent increase was not statistically significant and the percent increase could have happened anyway in the absence of the $41 million project or in the presence of the $41 million project. But the percent was negligible, small, not worth the $41 million.
Same goes for the next two indicators. Percent of women married who are currently using or whose sexual partner is using at least one modern contraceptive method. And the third indicator, percent of children who receive minimum dietary diversity and minimum meal frequency disaggregated. Right? Good indicators, global indicators. Just got to make sure that the $41 million that Canadian taxpayers spent resulted in a statistically significant different
increase due to the project, right? And finally, the ultimate outcome, same issue. This is a global indicator. Improved nutrition, nutrition -related rights, and gender equality for the poorest and most marginalized, especially women. adolescent girls and children under five years of age in Bangladesh, Kenya, Somalia, and Tanzania. Right? Three indicators? Very popular, common indicators. Global indicators. Percent of households achieved gender equality. Got some funky measure of gender
equality. Nothing wrong with the indicator, but we've got to show the percent. has gone up statistically and significantly right in the pmf same with anemia prevalence among adolescent girls and the third indicator percent of stunted children disaggregated by sex and country so it raises the overall issue of okay let's hope that they can show that the 41 million dollars was statistically significantly producing these increases in these percents in the right direction So the $41 million
really did have impact, right? So that's what we want to look at. And even when the indicators are global and therefore we know their proper valid measures of the outcomes where you just copy and paste, it's all the other outcome indicators and the analysis to make sure that it's statistically significant. That's assuming that the indicator, and as I've mentioned here, Many of the indicators are not valid measures of the outcome. So I've led to the conclusion, and I think I can add
three more now. I can cheat a bit. I can say that out of the 38 indicators, 10 of them are really good, valid outcome indicators, not seven, because I added those three. And the remaining, what is that, 28? Not good for the reasons of statistical significance or for other reasons
that I stated previously. So now this will go off, this episode recording with the PMF plus the PMF critique that summarizes all of the indicators, whether they're valid or not, will go off to the Secretary of State for International Development, along with the shadow critics for the conservative and NDP parties and the Bloc Québécois. And with the recommendation that these improvements be made. It's a judgment call as to whether or not the 41 million should be stopped. We don't know,
but I'm not going to go that far. And also, I'm trying to think now. Yes, we're going to invite World Vision Canada to come on to the podcast if they want and respond to my critique of their PMF. Thank you for listening. And we'll be back soon as I have another performance measurement framework. from another Canadian organization. So my next episode will follow shortly. Bye for now.