Improving Development Evaluation Podcast. I'm your host David Wand and welcome to episode seven, part one, where we introduce you to another new international development organization. And that international development organization for episode seven is Equitas, E-Q-U-I-T-A-S. Their headquarters is in Montreal, Canada, and you can learn more about Equitas at www.equitas.org.
The name of the project that they have been provided with funding to the tune of $18,280,395 from the taxpayers of Canada is called Achieving Equality Through Human Rights Education. And they're delivering this project in Burkina Faso, Haiti, Kenya, Senegal, and Tanzania. But they're also spending 35% of that 18 million in regions outside of those five countries, including north of Sahara in Africa and south of Sahara, as well as about 20% out of that 35%.
And then percent in Asia and 4% in Americas. In this project, there were 10 outcomes. And for each outcome in the performance measurement framework, they had two outcome indicators for a total of 20 outcome indicators, measuring whether or not they achieved those 10 outcomes.
And as I've mentioned before, if any of you are interested in receiving the performance measurement framework for this project or any other project that I've covered so far in the podcast, as well as the Excel summary of the outcome indicators for the project, feel free to email me at evaluatecanadaaid at gmail.com. I'll put that in the episode notes.
So before we begin a brief description of the project in terms of the services they deliver, I'll jump to the chase here, cut to the chase and tell you that none of the 20 outcome indicators adequately measure the outcomes. So Equitas cannot make the claim that they're achieving the outcomes in their performance measurement framework simply because the way they're going about measuring them is inadequate.
So with that summary, we're gonna move on to a brief description of the services that this project entitled Achieving Equality Through Human Rights Education, what services they're delivering. They have approximately nine target groups in the project.
So starting with the first target group, women's organizations, they receive quote, support to participate in events on advancing gender equality, the problem there is we don't know what they mean by support, there's no clear description of what they mean by support. Women's organizations also receive training on engaging duty bearers to fulfill their human rights obligations to advance gender equality.
Women's organizations also receive support in their continued engagement with duty bearers on the advancement of gender equality, but again, it's not clear in their performance measurement framework what exactly they mean by support. Women's organizations also develop action plans to advance gender equality.
They also participate in a number of joint awareness activities and they also share globally good practices in human rights based approach in gender equality, they share those globally and they also do context analysis related to gender equality. The second target group are called duty bearers and they are interviewed when women's organizations contact them on their advancement of gender equality. They also develop action plans to advance gender equality.
They also receive training on integrating a human rights based approach into their work to advance gender equality. Local structures are also supported to become more inclusive of women and while that is going on, the duty bearers are interviewed. Also duty bearers attend community forums that are promoting gender equality and women's participation. They're also involved in promoting gender equality through community actions and they also are involved in the context analysis on gender equality.
A third target group are human rights organizations and they participate in a number of joint awareness raising activities for advancing gender equality. They also are sharing globally good practices in human rights based approach in gender equality.
A fourth target group are civil society organizations and they are supported by the project to participate in alliances to advance gender equality but again, we don't know exactly what that support entails and in a good performance framework, measurement framework for outputs which these are, you would think they would quantify this by figuring out what they mean by support so that it could be better monitored. A fifth target group are partners, beneficiaries and intermediaries.
They're also involved in the context analysis on gender equality. They're also involved in sharing globally good practices in human rights based approach gender equality and they're also participating in the community forums.
A sixth target group are participants and these participants participate in two training sessions, one at the regional level held in East and West Africa on human rights training but also taxpayers be aware, there's one being held in Canada on training on human rights and according to the performance measurement framework, 461 people are gonna show up and if you do this math, that's about 1500 bucks for an airfare, I mean Haiti's probably a bit cheaper but return airfare from Africa to Canada
is about 1500 Canadian, multiply that by 461 and that lovely little human rights training session in Canada is costing you $691,500 Canadian dollars. That's quite an expensive output. Number seven target group, leaders of local structures. They also are supported to become more inclusive of women. Problem is again, they don't provide detail exactly of what they mean by support and if you look at the indicators, it's not clear how many sessions or what they're doing there.
The eighth target group are the partners in the countries, the developing countries and they receive organizational capacity building, training from Equitas and finally target group number nine are coaches and they are trained on community mobilization and human rights education to coach local organizations. I think they mean the local structures but again, it's not clear. The funny thing about this is even though the coaches are trained on this, after the training, everybody goes home.
There's no indication that the coaches are measured on whether or not they've learned what they were supposed to learn in the training. So that's a brief description of the services that the project is delivering.
Now I thought I would just give you just two examples because in part two, we're gonna go through all 20 outcome indicators with another evaluation expert and we're also going to invite Equitas to attend part two to respond to our critique because they may have some valid explanations as to our critique. So let's start with the outcome increased commitment of select women's organizations to engage duty bearers in dialogue on advancing gender equality in target countries.
One of the indicators for that outcome is number of plans implemented by select women's organizations to engage duty bearers in dialogue on advancing gender equality in target countries. Now the problem with that indicator is, and they even mentioned this in the performance measurement framework for another indicator, that they actually are concerned about quality criteria but they don't put it in this indicator. So the number of plans is not adequate. I mean the plans could be garbage, right?
You need to have some sort of minimum quality criteria where you can say that these plans actually show that the women actually indeed are committed and they've made effort to produce high quality action plans to advocate for gender equality, right? Even better, a better indicator would be measure the percent of plans that achieve quality criteria to engage effectively.
The other thing that's interesting, and you wouldn't know this unless you saw the performance measurement framework, is that in this outcome and in this indicator, they also interview the other side, that is the duty bearers that these women organizations go to to engage in dialogue on advancing gender equality.
So I would say forget the women, don't interview them, don't measure them, or if you're gonna measure them, measure the quality of their plans, go straight to the duty bearers and ask them, not the women's organizations, those women organizations that contacted you, was their engagement with you of sufficient quality to advance gender equality? Do you think they knew what they were doing? What they were talking about? Did they do their research, et cetera, et cetera?
You could go to the duty bearers instead. Now, the interesting thing is they interview the duty bearers, but we don't know if that is of any use, because the two indicators here for this outcome, they both talk about the women's organizations. They don't talk about the percentage of duty bearers, for example, who report that the women who engage with them in dialogue on promoting gender equality were of sufficient quality in their engagement.
Neither of the outcome indicators even mentions that, that they're gonna measure their duty bearers, but it does say in the PMF that they're gonna interview them. So it makes you wonder, why are they interviewing them if they're not included in either of the indicators that they're using to measure the outcome? So that's one example where the outcome indicator is flawed. It doesn't adequately measure any increased commitment.
And even if it did, it would be better to go to the duty bearers as a more objective measure, because you may get some self-reporting bias from the women organizations. So let's go to the next one. The next outcome statement I'm gonna look at is increased capacity of select women's organizations to engage duty bearers in better fulfilling their human rights obligations to advance gender equality in target countries.
And the indicator there is level of perceived capacity on a four point scale of select women's organizations to engage duty bearers in fulfilling their human rights obligations to advance gender equality. So we're talking about capacity to lobby these duty bearers. But perceived capacity, as we've mentioned several times on this podcast and other episodes, it's self-reporting bias. The women, of course, are gonna say, sure, I'm much better in lobbying than before this project started.
It's not a valid measure of technical capacity. And even if it was, even if we keep it, and their perceived levels of technical capacity to engage with these duty bearers has gone up, we still cannot say it was because of the project, because there's no comparison group. So what they should do is find other women's organizations in those countries that are not part of the project and ask them the same question over time.
Then, and only then can they show that the perceived capacity has gone up more in their project compared to other women's organizations who answer the same question but have nothing to do with the project. So that's the major problem there. So thank you for listening to part one featuring Equitas. Stay tuned for part two, where I'm going to send them part one here, this episode, part one.
And also I'm going to send Equitas the performance measurement framework, which they already have, as well as the Excel summary sheet that shows for each of the 20 outcome indicators why they are not properly designed to measure the outcomes. And an evaluation expert will also be exchanging, taking turns, going through the outcome indicators, explaining why we have issues with their outcome indicators.
But we hope that Equitas will accept our invitation and provide an opportunity to respond to our critique. Thank you for listening and stay tuned for episode seven, part two featuring Equitas.