I'm your host David Wand and welcome to episode six, part one. In this episode six, part one, I'll be introducing you to a new international development organization and in particular, one of their projects. And giving you a summary of the issues with that performance measurement framework, along with a description of the services that the project is delivering. So that new international development organization is colleges and institutes Canada.
You can learn more about this international development organization at their website, www.collegesinstitutes.ca. The project is entitled, A Thousand Women. I am woman, I exist, I participate. The cost of this project to the Canadian taxpayer is $18 million. And this project services are being delivered in Senegal, in West Africa. I've acquired the performance measurement framework for this project through an access to information request, because currently it is not available to the public.
Although I am advocating for these performance measurement frameworks to be put on the government of Canada project browser website. Before I describe the project in detail as to what the services are they deliver, I will come to the conclusion that of the nine outcomes in this project, there are 17 outcome indicators and only one of those outcome indicators properly measures whether or not the outcome has been achieved.
Therefore, I have led to conclude that this performance measurement framework in its current design is unable to support any claim by colleges and institutes Canada that this project is achieving its outcome.
So briefly, before I get into the description of the project, if you wish to receive the performance measurement framework for this project, as well as the summary of the outcome indicators, all 17 and how each one of them fails to properly measure the outcome for the project, you can email me at evaluatecanadaaid at gmail.com. And I'll make sure to put that information into the notes that are associated with this podcast, part one, episode six.
So for this project of $18 million, there are 11 target groups that receive a variety of services from this project. So let me go through them with you. The first target group are male leaders who receive training in leadership. The second target group are vulnerable women and they are mentored by women leaders. And they're also trained by women leaders on leadership.
And they also have partnerships established and support measures to support their employment, integration and business development. They also receive training on female entrepreneurship. The third target group is male leaders and women leaders who are also trained on leadership. The fourth target group is communities. They receive campaigns that promote, I quote, gender equality. The fifth target group are associations of entrepreneurs and business people.
For this target group of associations and entrepreneurs and business people, they are promoting to this target group, quote, the participation of women. The sixth target group, they refer to as target entrepreneurs. They refer to as target audiences. To these audiences, awareness campaigns are delivered to, quote, raise awareness of the importance of women's participation in all decision making bodies. The seventh target group are regions.
And in these regions, advocacy actions, quote, to promote women as full partners in the development of communities are reached. Target group eight are the actual project staff and they receive training to support the women target groups in accessing the project services. That is access to training on leadership, entrepreneurship and integration into employment. The ninth target group are civil society organizations, women's groups and economic interest groups.
They receive a consultation mechanism between these three groups that is put in place and is functional. The 10th target group are government partners where they also receive a consultation mechanism between these government partners that is put in place and is functional. And finally, the 11th target group are teachers and directors of training centers. These teachers are trained on the mechanism for validating acquired experience. They're also trained on the, and I quote, gender approach.
So that gives you an idea of the description of the services that are delivered under this project, as well as the target groups that receive these services. And as I mentioned earlier, when we reviewed the 17 outcome indicators, only one of those 17 outcome indicators was designed in such a way that we could claim that the outcome indicator for the outcome was properly measuring that outcome.
So for part two, I'll be inviting an evaluation expert to attend to go through those outcome indicators, at least nine of them. So we make sure we cover all nine outcomes and critique each outcome indicator to explain in detail how it fails to properly measure the outcome. We will also be inviting a representative from colleges and institutes Canada to attend part two of the podcast to respond to our critique.
And just to reiterate, there are nine outcomes for this project from the performance measurement framework, and there are 17 outcome indicators that attempt to measure the achievement of those nine outcomes. Before I finish, I'm going to look just at three outcome indicators and explain why they fail to properly measure the outcomes. So one of the outcomes in the performance measurement framework is, and I quote, strengthening the empowerment of women among the most vulnerable across Senegal.
And they use what's called the female empowerment framework. What is fantastic about this measurement is they actually do use a comparison group that is women who are not in this project receiving that spectrum of services that I described to you earlier. And they're also measuring their levels of female empowerment before the project starts, as well as after the project starts. However, there are two problems with the measurement of women.
Even though they're using a comparison group, which is a good sign. First of all, they're only measuring it once per year. So this means, as I mentioned in my trailer, it would be very easy for the project women to achieve their outcomes by themselves in between the trainings that they receive in the project. Now, you could compare that with the comparison group. But it would be better if they did it more frequently.
But the second reason there's a problem with this indicator, even though they're using a comparison group, is the female empowerment index looks at six dimensions. They are one, violence against women, two, employment, three, education, four, reproductive health care, five, decision making, and six, access to contraceptives. Now, for this project, there's clearly a focus on employment and decision making.
The problem is, when you look at the project services related to employment for these women and the services related to improving their decision making, there are two indicators in the performance measurement framework. Related to these two areas, the first one is percent of women in employment or self-employment after training. The problem here is there's no comparison group there for that service being delivered, the training.
Also, there's no measure of their incomes after the training, which would be a better measure of whether or not, even if they do get employed, are they making any money? Are they making any money from that employment? And is it due to the project training that they receive, for example, in entrepreneurial skills or how to set up a business? So they didn't use a comparison group there.
So we need to know for the female empowerment index, did they also ask that comparison group for the project women, by the way, you attended some training on entrepreneurship. Did you earn any income after you received the training?
It's not clear if they actually asked them that because we don't know if the same two comparison groups, that is the project women and the non-project women for the female empowerment index, if they're also using that same group of women for those two groups for all the other parts of the project, such as training them in entrepreneurship. We hope they are, but we don't know.
It's not clear because in the performance measurement framework, it doesn't show a comparison group on this indicator of percent of women in employment, self-employment after the project training. The other indicator that they use that's related to decision making is the percent of women targeted by the project who participate in decision making bodies in their communities. The problem here is it is only measured once per year.
So again, these women could easily increase their participation by themselves without the project training. There also is no comparison group at the time that they're receiving training in decision making. A better measure of whether or not they're participating in decision making is not to ask the women themselves because they're self-reporting bias. The project women, of course, are going to say, yes, social desirability bias.
Yes, I have increased my participation in the community in decision making bodies. A better measure would be to see if these women have increased their membership on decision making bodies, either municipal, elected, or state. Either municipal, elected councils, or school boards, or whatever they consider to be a decision making body. Have these project women actually become members in positions of power where they can make decisions.
That would be a better objective measure of whether or not they've increased their participation on decision making bodies. So that's the problem there. Even though the female empowerment index, they're using comparison groups at the top, at the end of the project. We don't know if all the trainings they received during the project are making a significant difference compared to the non-project women because they were not using comparison groups.
Now, hopefully, College and Institutes Canada will attend part two of the podcast and will explain perhaps, maybe when we were using the female empowerment index, we were also using these other questions on these other indicators at the same time.
And then we could show that even when the project was delivering these services, the women were also reporting on their incomes, their employment levels, also to show that their employment was going up and their ability to participate in decision making was also going up compared to the non-project women.
So then now that you have a brief description of the project, as well as my conclusion that the outcome indicators, at least three of them that I've talked to you about there, are not properly designed to show that the outcomes were actually achieved for the project.
Now, we're going to send this part one to College and Institutes Canada to invite them to attend part two and there will also be an evaluation expert from Sweden who will be attending the podcast to take turns with me in explaining to College and Institutes Canada why we believe the outcome indicators are flawed, but more importantly, we're also going to suggest how these indicators could be improved so that the overall evaluation of the project could be improved.
Thank you for listening and stay tuned for episode six, part two, featuring College and Institutes Canada and the flaws in their performance measurement framework and how we can improve those indicators leading to a better evaluation of the project.