Welcome to the Improving Development Evaluation Podcast. I'm your host David Wand, and welcome to episode four, part two, where we continue our review and critique of the Ghana Rural Integrated Development, that's GRID, G-R-I-D, International Development Organization, and their project in Ghana entitled, Improving the Well-Being of Rural Women and Girls in Ghana.
The cost to the Canadian taxpayer for this project is $1,337,797, and it has 18 outcome indicators covering seven outcomes that we will review each of those 18 outcome indicators in this part two. On January 7th, I sent an email and also left a voicemail with GRID, inviting them to attend the podcast. I did not receive a reply.
Then I followed up again inviting them also again on January 18th with an email and voice message, but so far have not heard a reply regarding my invitation for them to attend this podcast in part two, to respond to my critique. So I will go through the 18 outcome indicators covering the seven outcomes, starting with the first outcome, which is increased ability of responsibility bearers to protect the rights of women in Northern Ghana. For this outcome, they have two outcome indicators.
The first outcome indicator is number of times strategies have been implemented by responsibility bearers to protect the rights of women. And the second indicator is the percent of women who are fairly or very confident that community leaders are protecting their rights. Now the problem with this indicator is the number of strategies that have been implemented by responsibility bearers to protect the rights of women is quite a good indicator.
You would think that as the strategies go up, this shows an increased ability to advocate. The problem is there's no comparison group outside of the project of other responsibility bearers to see how well they are protecting the rights of women in Northern Ghana. So that was the problem with that indicator.
The second indicator, percent of women who are fairly or very confident that community leaders are protecting their rights is also quite a good indicator because you would expect as confidence levels go up in these women, in their perception of the increased ability of these responsibility bearers, this would seem to be a good measure of their increased ability to protect them. But unfortunately again, there's no comparison group.
So we don't know if responsibility bearers in another part of Ghana outside of the project could also be doing quite well. So that was the problem with these two indicators for that outcome. The next outcome is improved ability of women from 30 rural communities to advocate for their wellbeing. And there's three indicators for this outcome. The first indicator is the number of women farmers able to successfully advocate for long-term access to land.
The second indicator is the percent of women who feel they are completely or mostly able to advocate on behalf of women's concerns. And the final indicator is percent of women co-op leaders able to identify one or more things they plan to lead their co-ops and or communities in doing to improve quality of life. The problem with this, the first indicator, able to successfully advocate for long-term access to land is that it doesn't directly measure the ability to advocate.
A better measure would be just to look at the amount of land that they actually acquired for the women. If they're advocating for access to land, instead of advocating for access, successfully advocate, they should just look at how much land they were able to acquire for these women. You could even argue that it's not related, the indicator in its current form. So that would be a better way to improve the indicator to actually measure their improved ability to advocate.
And that is actually getting the land for the women. The second indicator, percent of women who feel they're completely or mostly able to advocate on behalf of women's concerns, they're asking for feeling. And instead, as I've mentioned before, they should measure directly their ability to advocate, more objectively than just asking them if they feel they are completely or mostly able to advocate. It's not a proper objective measure of the ability to advocate.
The third indicator, the percent of women co-op leaders able to identify one or more things they plan to lead their co-ops or communities in doing to improve quality of life. They measure this through focus group discussions and they only do it three times over four years of the project. This is not going to show that the training they received to advocate is actually responsible for their ability to advocate. There's no objective measure there.
It's a focus group discussion and they don't do it frequently enough. They only do it three times over the four years.
So again, a better measure of their ability to advocate would be to test them on their ability to advocate and maybe do that every six months and even better compare it against another group of women who are advocating outside of the project and see on the same measures, maybe a test or reviewing their advocacy products that they deliver, such as maybe a pamphlet or a radio announcement where they're advocating for these women's rights and they could compare the quality of the products
they've produced against another group of women and rate and compare the quality of those products objectively. The next outcome is improved access to senior secondary and post-secondary education by young women in 30 rural communities. The two indicators for this outcome are the number of women less than 25 years of age attending secondary and post-secondary school and the number of women versus men completing senior secondary school by age 18.
Now, these outcomes of improved access actually are quite straightforward because they're just trying to deliver more education to these women through cash to pay for their school fees or scholarships to pay for their post-secondary tuition. So this is pretty straightforward in terms of improving access to education. This would be considered to be good measures. The problem is it's more a service through cash and scholarships rather than expected outcomes after they've attended school.
The next outcome is improved knowledge and skills in managing sustainable, resilient, and productive agricultural livelihoods by adult women in 30 rural communities. Here we have three outcome indicators for this outcome. The number of women able to produce nine bags of peanuts on one acre farms. Second indicator is the number of women owning one or more healthy female goats.
And the third indicator is the percentage of women who are moderately or highly confident in their knowledge and skills in farm management for food and crop production. And the percentage of women who are moderately or highly confident in their knowledge and skills in livestock rearing. Now, I mentioned this in part one, they're getting close, number of bags of peanuts on one acre farms, it's pretty good, but they haven't weighed the bags, they could be empty.
So we wanna get a third party to get in there and actually weigh the bags of peanuts. So we have the total weight of peanuts produced, which would be a good measure of their ability, skill and knowledge levels in growing peanuts, which is good. And even better, it should be the median weight of peanuts. And they should compare that against another group of women peanut farmers that did not get access to the training through this project.
And hopefully you'll see the median weight of this group through the project having a higher median weight than the women outside of the project who did not receive any peanut farm training. And the same goes for the goats. You would wanna get the median number of goats and compare that with another group of women who are raising goats, but haven't had the opportunity to receive training on how to raise goats.
And finally, the last outcome indicator is the confidence levels in their knowledge and skill levels. I would say you don't need that. If you're directly measuring their ability through how many peanuts in weight they produced and how many goats they have produced and sold, that would be sufficient. There's no need to ask them on their confidence levels regarding their knowledge and skills. So you wouldn't even need that third indicator.
The next outcome is improved participation in equitable household and community decision-making by women in 30 communities in rural Northern Ghana. We have three indicators for that outcome, and they are number of community members reporting that women participate in key household decisions. Second indicator, number of community members who are satisfied with women's participation in community decision-making.
And number three, the third indicator is the number of girls completing or planning to complete secondary school and post-secondary education. So in the first one, community members reporting that women participate in key household decisions, that's a bit of a tricky one because you're asking community members to report on another group of women. That's the women that actually are inside of the household. That is kind of questionable.
I would rather survey the women themselves inside their households to see how they feel they're doing when it comes to participating in key household decisions. So there's the flaw there, there's not a direct measurement. The second indicator, community members who are satisfied with women's participation in community decision-making, that's a pretty good measure.
They could improve it a bit by actually looking at the actual representation of women on these community decision-making bodies, such as boards, school boards, for example, or even municipal elected councils, to see the percentage of women that are on these boards. These are bodies, community decision-making bodies that actually have power to make decisions. As for the third one, I would take it out.
Percent of girls completing secondary school or post-secondary school, yes, it's in a measure of improved participation, obviously through attendance, but it's not related, the completion of educational school with the equitable household decision-making.
Although you could argue that the more girls that graduate from secondary school and post-secondary education, the more knowledge they have in terms of trying to make a decision in the household or at the community level, but the other two indicators, if improved upon, would be sufficient to measure improved participation. The next outcome is increased participation in sustainable productive work by women in 30 communities in rural northern Ghana.
Here we have three outcome indicators for that outcome of increased participation in productive work. They are the percent of women earning $500 or more per year through productive work, the percent of women under 25 completing or expecting to complete secondary and post-secondary education, and the third outcome indicator for that outcome is the percent of young and adult women who think they are somewhat or very likely to be hired for a job outside their household.
Now, the first indicator I've mentioned previously is excellent. They are trying to get at the incomes that are measured from learning how to grow and sell peanuts and learning how to raise and sell goats. So it's excellent. They're trying to get in the right direction. It just needs a slight correction, and that is instead of 500 per year, they should measure the median income of the 750 women in the project and watch their incomes go up.
It also should be specific, like earlier in the performance measurement framework, they should be measuring the median incomes for those growing peanuts and those raising goats. So it's not just productive work, it's precise, specific to the project training, which is on how to grow peanuts and how to raise goats and sell them, right? The second indicator, again, is talking about completion rates for women in secondary school and post-secondary education.
Yes, it's a good measure of increased participation in productive work because they will have the potential to increase their participation in earning a livelihood based on the benefit of completing their education. So yes, that is important. I think focusing on the incomes is sufficient enough, and I would eliminate that indicator.
You can keep it into track how many women as an output completed their education after getting the cash from the project or getting the scholarship from the project, that's useful. But in terms of productive work, I would look at incomes as a better measure of whether or not they've increased their participation in productive work.
The last outcome indicator there, dealing with the percent of women who think they are somewhat or very likely to be hired for a job outside their household, again, it's not a bad measure, but it's indirect. I would focus on the incomes. The next outcome is improved wellbeing of women and girls in 30 rural communities in Ghana's northern and Brong Ahafl regions. And the indicators there, they have, there's two indicators. The first one is percent of women living on $2 US or more per day.
And the second indicator is level of perceived wellbeing on a scale from one to 10 of the participating women. Now the first indicator is a standard, I think, World Bank measure. They don't have to do that. They can just, again, focus on the incomes over time for the project, for the training they have received. They don't have to stretch it and try to claim that there's this percent of women in the larger community living on $2 US per day. There's no connection. It's too far stretched.
They should just focus on the incomes of the women in the project to make sure that they have been going up. And ideally, they should compare it against women who are not receiving training on how to grow peanuts and how to raise goats and track their incomes also. And hopefully the project will have higher median incomes than those women not in the project. That will be a better way of evaluating the effectiveness of the training project that we're talking about today.
Now the second indicator, the level of perceived wellbeing of participating women, that's a pretty good indicator. I think it's okay. But if we look at the performance measurement framework in more detail, you'll notice it's only measured once per year over this four years of this project.
So it would be very difficult to claim that the perceived wellbeing is going up if it is in fact going up, that it's due to all this wonderful training that these women are receiving on how to grow and sell peanuts and how to raise and sell goats if it's only measured once a year. There could be other factors involved in why their perceived wellbeing is going up that has nothing to do with the training they received in the project.
So ideally what you'd like to do is measure it more frequently in the project and again compare it, this perceived level of wellbeing with other women outside of the project. Ideally those also involved in peanut farming and goat farming and then you could show that the perceived wellbeing is going up greater in the project compared to the women outside of the project who are also measured on their perceived wellbeing.
Now in addition to these seven outcomes and the 18 outcome indicators I've covered, there's also another eight outcome indicators that all have the same problem. They have pre-test before the training and a post-test just once before they get tested on the knowledge and skill levels of the content of the training and then they get tested once more afterwards. Now that's pretty good. They're at least doing some objective testing which is great.
The problem is they only do it once before and once after and as I've explained in the trailer, they need to show more frequent testing unless it's done during the same day. Now that's interesting. 750 women, they come in and do a test before the training and then are tested after the training and you could show that their skill levels have gone up from the test, that would be good. We don't know from the performance measurement framework if it's a same day training.
Ideally what you also want to have is a comparison group where they're measuring women who don't attend the training at all but are also given the same test before and after. That's ideally what you want to do so you also want to have a comparison group and those content trainings are in the following areas. Ability to identify justice issues specific to women along with an appropriate solution. Ability to identify key actions to maintain health.
Ability to identify best practices for safe food preparation. Ability to identify ways of successfully accessing markets and understanding of basics in sustainable peanut farming and small ruminant rearing. And knowledge of peanut farming and knowledge of small ruminant husbandry. I guess that's the goats. And knowledge of women's concerns.
So they do pre and post testing for these various subject areas but I think if they had comparison groups that would be better because then they could show that the project training is more effective increasing knowledge and skill levels in these subject areas compared to a group of similar women who are not in the project.
So now that I've finished my review and critique of all the outcome indicators for the project, if you want to email me at evaluatecanadaaid at gmail.com, I would be happy to send you the performance measurement framework for this project as well as the summary of all of the 18 outcome indicators.
And once again, if there are any evaluation experts out there who would like to participate in a part two of a podcast where we go through all the outcome indicators explaining how they do or do not fit properly with the measurement of the outcome, feel free to email me and I'm happy to send you the remaining performance measurement frameworks along with my critique of the performance measurement framework so that you could participate as we have a lot more PMFs coming shortly.
Now what I will do is I will email this part two and part one to the minister for international development along with the shadow critics for international development for the conservative party, the new democratic party and the Bloc Québécois. And I will also recommend in that email that performance measurement frameworks immediately be available to the public on the Government of Canada, Global Affairs Canada project browser website.
And obviously also recommend that there be improvements in development evaluation by developing better outcome indicators to measure project outcomes. Thank you very much and stay tuned for episode five, part one where we will be introducing once again another project and the project that we will be doing in episode five, part one, cost to the Canadian taxpayer, $19 million. So stay tuned and thank you for listening.