Welcome to the Improving Development Evaluation Podcast. I'm your host David Wand and welcome to season one, episode four, part one. The International Development Organization where you're going to talk about in this episode, part one is the Ghana Rural Integrated Development International Organization or SHORTGRID, G-R-I-D. You can learn more about this International Development Organization if you check out their website at www.GRID.org.
Their headquarters is located in Milton, Canada and the Ghana Rural Integrated Development is doing a project called Improving the Well-Being of Rural Women and Girls in Ghana. And the cost to the Canadian taxpayer for this project is $1,337,797. Now before I briefly describe the services delivered for this project, I'm going to first talk about the performance measurement framework that I acquired through an access to information request for this project.
The PMF or performance measurement framework for this project has 18 outcome indicators measuring seven outcomes. And based on my review of this performance measurement framework, I have come to the conclusion that only two out of these 18 outcome indicators properly measure the outcomes. And therefore I cannot support GRID's claim that they have achieved their project outcomes with their project services delivered.
In a breakdown of this summary of the 18 outcome indicators, 11 of the outcome indicators do not measure the outcome at all, but something else not related to the outcome. Another three outcome indicators do measure the outcome, but they have no comparison group outside of the project. So that means women and girls outside of the project could have also achieved the outcomes at the same amount or even better than the project itself.
Another outcome indicator asks people how they feel about achieving the outcome rather than measuring objectively the outcome. And one other outcome indicator measures the outcome, but not frequently enough to claim that the services achieved the outcome rather than other factors. So that's my conclusion for the performance measurement framework.
And if you want to receive the performance measurement framework for this project or and or the summary of all of the 18 outcome indicators, feel free to email me at evaluatecanadaaid at gmail.com and I would be happy to send those documents to you. And also if there's any evaluation experts out there that would like to be on my podcast to also critique the performance measurement framework for upcoming episodes, feel free to email me.
I will be happy to send you those performance measurement frameworks that I have acquired through an access to information request. Now the services for this project, there were four target groups that these services were being delivered to. And those four target groups were number one, women. And the focus of the project was to train them on how to grow and sell peanuts and how to raise and sell goats.
In addition to that, they were also trained on safe food preparation, healthy behaviors and how to access markets to sell those peanuts and sell those goats. A second target group were women co-op leaders and they were trained in leadership, also peanut farming and animal husbandry and also in acquiring knowledge on women's concerns. And a third target group were women and men and they were trained in household financial management.
And a fourth target group were girls and young women who received cash and scholarships at the secondary school level and at the post-secondary school level. So those services for those four target groups when delivered were expected for the project to achieve one of seven outcomes. And those seven outcomes taken from the performance measurement framework are, outcome number one, increased ability of responsibility bearers to protect the rights of women in rural northern Ghana.
The second outcome, improved ability of women from 30 rural communities to advocate for their well-being. Outcome number three, improved access to senior secondary and post-secondary education by young women in 30 rural communities. Outcome number four, improved knowledge and skills in managing sustainable, resilient and productive agricultural livelihoods by adult women in 30 rural communities.
Number five, improved participation in equitable household and community decision-making by women in 30 communities in rural northern Ghana. Outcome number six, increased participation in sustainable productive work by women in 30 communities in rural northern Ghana. And outcome number seven, improved well-being of women and girls in 30 rural communities in Ghana's northern and Brongahafu regions. So there you have the outcomes and the services.
Now, for this part one, I'm just going to give you three examples of some outcome indicators for some of the outcomes in the project where they run into these various problems of poor measurement or no measurement at all. And despite the small size of this project, only over a million dollars if you've been following the podcast, the other projects I've been reviewing go, you know, anywhere up from five million dollars up to 37 million dollars.
But in this example, I want to give some credit where credit is due. And as I'll get into detail here, you'll notice that some of the indicators are quite good as they start to actually measure incomes for expected immediate outcomes or longer term outcomes for training on how to grow and sell peanuts and how to raise and sell goats. So we'll get right into it now. So let's take a look at the outcome.
Number four, improved knowledge and skills in managing sustainable, resilient and productive agricultural livelihoods by adult women in 30 rural communities. One of the outcome indicators for this outcome number four is, and I quote, number of women able to produce nine bags of peanuts on one acre farms. Not bad. They're getting close.
If you learn how to grow and sell peanuts, eventually you're going to make income from selling those peanuts, but before you make the income from selling the peanuts, you have to be able to know how to grow and sell the peanuts. And so they're actually getting close by saying, let's take a look at the bags of peanuts that you've produced, which is pretty good. Only problem is those nine bags of peanuts could be empty.
So I would suggest it's going in the right direction, this outcome indicator, but ideally what you'd like to do is weigh the bags of peanuts to see the weight of the peanuts that they've actually been able to harvest.
So I would change the number of bags to the weight in kilograms, and please change the number of women to the median weight of peanuts produced by these women, along with the number of women in the project that were producing the peanuts that are associated with the median weight in kilograms of peanuts produced.
The interesting thing about the performance measurement framework is if you take a closer look at it, you'll notice that the project staff are responsible for coming up with how many women produced nine bags of peanuts. Slight conflict of interest there. They review the project harvest records, as it says in the performance measurement framework. That should ideally be done by a third party. They should just go in and go to the women and say, show me your peanuts. Let's weigh them.
That's a more accountable and transparent way of coming up with this weight in kilograms of peanuts produced. So that's a good indication that this organization, GRID, is going in the right direction. They're saying, show me your peanuts, but you want to get the weight, ideally. That's what you'd like to do. So the second outcome, outcome number two, improved ability of women from 30 rural communities to advocate for their well-being.
The indicator here, one of them, is the percent of women co-op leaders able to identify one or more things, quote, they plan to lead their co-ops and their communities in doing to improve quality of life. The problem with this indicator is the plan to lead does not measure improved ability to advocate. It's too soft. It's too indirect. Also, a plan to lead is not advocating. The performance measurement framework shows this is done through focus group discussions with these women co-op leaders.
That's not good enough. What you could do is, since you're training these women co-op leaders to be technically able to advocate, you should test them objectively on their ability to advocate. So for example, you could look at some advocacy or communication documents that they've produced and score them.
And you could also look at their presentation skills, because I presume that they have to advocate to somebody, maybe the government, local government, people, governments that are going to respond to the interests of these women. So you have to see what their presentation skills are also. These are just examples. But the point is, a plan to lead is not good enough.
If you're training them on how to advocate, you're training them how to lead, then you should objectively measure their ability to lead and their ability to advocate. So you could check out their documents that they're producing for advocacy on their quality. And you could check out their communication presentation skills on advocating. And you could score those.
In addition, even if their technical ability to advocate does go up after the training, you need to have either frequent measures to show that, or a comparison group of women who've been not trained by the project, but are advocating for women in some other part of the country that are not in the project, because you would expect women who've been trained in this project to have better levels of advocacy skills. The same goes for, back to the other outcome, for the weight of the peanuts.
You would expect women peanut farmers in Ghana, for example, outside of the project, to have lower median levels of weight of peanuts compared to the project where they were actually training them on how to sell and grow peanuts. So it would be helpful also to have a comparison group there too. Another example is outcome number six, increased participation in sustainable, productive work by women in 30 communities in rural northern Ghana.
The indicator here is the percent of women earning at least $500 per year through productive work. As you can tell, the indicator earning income has got nothing to do with increased participation. Now you could argue that if they are increasing their participation, eventually their incomes are going to go up from doing that agricultural work, which is fine.
And in fact, it's an excellent start for this organization to start measuring incomes, rather than asking them how they feel about their incomes, actually starting to measure.
The only thing I would suggest here is, once again, like the number of bags of peanuts, here instead of just measuring the percent of women who earn $500 or more per year through the selling of peanuts or the selling of goats, they could simply get the median income for each of these activities for selling peanuts and for selling goats. But it's still useful because they're moving in the right direction by looking at income.
And also again, they should have a comparison group of women outside of the project who are also selling peanuts and raising, selling goats to see if their incomes, their median incomes are lower. And that in fact, what we would do is instead of the percent of women, you would just have the median income for these women and the median income for another group of women in the country that have not been part of this project and compare their incomes.
Because you would obviously expect the project training to be better at training them and their incomes would go up and that those income levels would be greater than the median income of the women who do not have the benefit of the training from the project. We don't know as you know, if that's going to be the case unless you have a comparison group. So those are just some examples of where the outcome indicators do not properly measure the outcomes.
So what I'm going to do now is share the summary of all of the 18 outcome indicators along with the performance measurement framework with the representative from the Ghana Rural Integrated Development Organization and invite them to respond to this critique of part one. We'll make sure that we go through at least seven outcome indicators covering the seven outcomes in the project.
But ideally we'd like to go through all 18 to explain why they believe or why they disagree with these indicators being a problem in supporting their claim that they've achieved their outcomes.
And then once we finish part two, we will send it to the Minister for International Development as well as the shadow critics for the opposition party and the third party recommending that again immediately the performance measurement framework for this project and all future projects for this organization and for all organizations for that matter be included on the global affairs Canada project browser website so that everybody can see that these
performance measurement frameworks actually show the flaws that we've discussed. And ideally the second recommendation would be to begin starting to properly measure the outcomes with better outcome indicators such as three examples I have provided in this part one. Thank you for listening and stay tuned for episode four part two featuring the Ghana rural integrated development international organization.