Welcome to the Improving Development Evaluation Podcast. I'm your host David Wand and welcome back to Episode 7, Part 2, where we continue our discussion of the international development organization called Equitas that has their headquarters in Montreal, Canada, and we are talking about their project worth about 18 million Canadian dollars, courtesy of the Canadian taxpayer, and this is a human rights project where they're promoting women, gender equality in five different developing countries,
and we're going to go into more detail about their performance measurement framework for those five developing countries, and they are Burkina Faso, Haiti, Kenya, Senegal, and Tanzania, but they also use some of the 18 million dollars for promoting gender equality and human rights in certain regions such as West Africa, and we won't go into detail about Part 1, we've already done that. We're going to focus on the outcomes as we usually do in Part 2.
Just to give you some background, on April 23rd, I sent Equitas their performance measurement framework along with our critique of their performance measurement framework that outlines why each of the 20 outcome indicators do not adequately measure properly the 10 outcomes in the project.
Then I also sent them the Part 1 of the podcast along with the PMF and the Excel summary sheet, and then on April 29th, I received an email back from Equitas where they declined to attend Part 2, which is today, of the podcast to respond to our critique of their performance measurement framework, and they also declined to provide a written response as to what their response would be to our critique, which is unfortunate, but we will proceed as usual
with Part 2, going through each of the 20 outcomes, explaining why they fail to properly measure the outcomes. And with me today is an impact evaluation specialist from London, Ontario, Canada. Her name is Yvonne Okéké. She has a master's degree in impact evaluation for international development from the University of East Anglia in the United Kingdom. Welcome, Yvonne. How are you? Thank you, David. I'm doing absolutely fine. How are you doing?
I'm fine, as usual, continuing our quest to let people know that we've got some issues with some international development projects. So we will start, as usual, with the 20 outcome indicators covering the 10 outcomes, and I'll start with the first outcome of this performance measurement framework, which is increased commitment of select women's organizations to engage duty bearers in dialogue on advancing gender equality in target countries.
And the outcome indicator I'm looking at is, and I quote, number of plans implemented by select women's organizations to engage duty bearers in dialogue on advancing gender equality in target countries. The problem with this outcome indicator, it doesn't adequately measure increased commitment to engage with the duty bearers because it's just reporting on the number of plans. And the problem here is the plan has to be of sufficient quality to engage with these duty bearers.
Just reporting on the number of plans is not adequate. The plans could be garbage. We just don't know. And to be fair to the Equitas, maybe somewhere in the performance measurement framework, but I cannot find it. And I want to repeat to the listeners out there, if you want a copy of the performance measurement framework from Equitas, as well as our critique, feel free to email me at evaluatecanadaaid at gmail.com, and I would be happy to send it to you.
And you're also welcome to ask for any of the other performance measurement frameworks and any of the other critiques of those performance measurement frameworks that we've covered in the podcast so far. So that's the issue here. The number of plans is not adequate as a measure of increased commitment. What they should do instead is measure the percentage of plans that achieve, quote, quality criteria to engage effectively.
In fact, another indicator in the performance measurement framework that we'll discuss later actually talks about quality criteria, but they don't seem how to do it here. Also, a thing to note in the performance measurement framework for this indicator is they also interview the duty bearers, the ones who receive these women who are lobbying them to promote gender equality.
And another recommendation here would be to ask them, not the women's organizations, but go to the duty bearers and ask them, do you think these women who are engaging with you on advancing dialogue on advancing gender equality, are they doing it adequately? Because the whole project's based on training them on making sure that they can effectively lobby these duty bearers. So that's the end of that one.
So what I'm going to do now is read the next outcome statement and Yvonne's going to look at the next outcome indicator. So here the next outcome statement is, and I quote, increased capacity of select women's organizations to engage duty bearers in better fulfilling their human rights obligations to advance gender equality in target countries. And the outcome indicator that you're going to look at, Yvonne?
That is the percentage of intermediaries, men and women, trained in transferring knowledge to select women's organizations on engaging duty bearers in fulfilling their human rights obligations to advance gender equality in target countries. That is the outcome indicator that has been specified. And this indicator is flawed in that it has the disadvantage of self-reporting bias.
The women's organizations are expected to have increased technical capacity to engage or lobby the duty bearers based on the training already provided in the previous output. Hence, they should be tested on this technical ability, just knowing that the knowledge has been transferred is not really good enough for this particular outcome. That's right.
And there's even a weird, strange intermediary of the women and men who are trained that then transfer the knowledge to select women's organizations who then engage with the duty bearers. So it's kind of strange. They're just looking at the percent who are able to transfer. Now again, at the performance measurement framework, maybe they should have a little sentence where they say, see the following measurement tool, but they don't.
So we can only assume that they've missed the mark here on properly measuring increased capacity. Yeah. So. Absolutely. Yeah. And the next outcome is increased opportunities for civil society organizations to collaborate on advancing gender equality in target countries. And the indicator for this is interesting. It's quote, type of opportunities available for civil society organizations to collaborate on advancing gender equality in target countries.
Now initially when you read that, you think, well, if civil society organizations have increased opportunities to collaborate, that's a good measure, I would think. You know, when you first read it, you think that's a good measure because they can expand their opportunities to collaborate, which will over time be a benefit for the country in advancing gender equality. Makes sense.
The problem is it's not really defined and it should be connected to the technical capacity to engage and promote gender equality. So maybe they do have more opportunities. So yes, you could argue that's pretty good, but it should be connected to their ability technically to engage and promote gender equality. So what exactly will these civil society organizations do when they get the opportunity to engage to advance gender equality?
They can have all these wonderful opportunities running around, collaborating with other civil society organizations, but don't forget they have to engage. They have to promote gender equality.
So that's the problem there is on the surface, it looks good, but we need that type of opportunity to be clearly defined and assume that that type of opportunity is equated with better performance on promoting gender equality and then also making sure that that in turn is connected to a better technical ability to engage and promote gender equality. So I'm criticizing it, but it's still not a bad indicator to be honest, but it doesn't go far enough and they don't define it. Right.
So now we're going to move on to the next outcome, which is, and I quote, increased capacity of women's and human rights organizations to carry out human rights education to advance gender equality. And you're going to be looking at the outcome indicator. Level of perceived capacity on a four point scale on women's and human rights organizations to carry out human rights education to advance gender equality.
Now, a measuring capacity is not a bad idea, but level of perceived capacity of those you train is not a valid measure of technical capacity. If people have been trained on how to carry out human rights education to advance gender equality, what should be done to measure the particular outcome is to test them. They should undergo tests because the perceived measures would need a comparison group, which is not also provided for in this PMF.
There is no comparison group and also measuring level of perceived capacity for this particular outcome would not technically tell us whether they did a good job or not in increasing the capacity of women's and human rights organizations to carry out this education, human rights education to advance gender equality. So we've got two points there. First of all, the perceived capacity, measuring perceived capacity is not a valid measure of technical capacity.
And if they truly want to test, if a good job has been done in training these women's and human rights organizations, we need a comparison group. Thank you. Sure. Thank you, Yvonne. Yeah, that's right. Even if we say, okay, look, we'll let you do perceived capacity. You've got to show that the project has improved or increased that perceived capacity, even though it's self-reporting bias and they're all going to say, oh yeah, much better thanks to Equitas.
But maybe there's another group of women and human rights organizations out there that also are experiencing increased capacity to carry out human rights education to advance gender equality in the country. And they've had nothing to do with the project.
So yes, need a comparison group if you want to show that your project of $18 million actually was involved and responsible for improving, increasing that perceived capacity, even though we know it's not a valid measure of the ability to carry out human rights education. Right. So the next outcome is increased opportunities for women and or girls to participate in decision-making process within local structures in target countries.
Here the outcome indicator measure is level of confidence on a four-point scale of women and or girls in their ability to participate in decision-making process within local structures. Again, we have a self-reporting bias in their ability to participate. That's one issue. And the other issue is maybe Equitas is going to say, look, we're not necessarily interested in just increasing technical ability to abdicate.
We also want to increase the confidence levels of women or girls in their ability or just their confidence to speak up and participate. And I think that's a fair point. Problem is they don't have a comparison group. So they can't even show if their project increased perceived levels of confidence reported by these women or girls. So they can't have it both ways.
If they don't want to, if they want to measure technically increased opportunities, for example, on a decision-making body, do they really have the power to make decisions, which they're not doing here? Or if they want to stick with level of confidence, then again, they have to show their project is doing a better job of increasing levels of confidence than women and girls in the country that are not in the project.
So next outcome is increased capacity of intermediaries that we mentioned earlier, women and girls, to mobilize communities to advance gender equality in target countries. And you're going to be looking at an outcome there, Yvonne. Yeah, the outcome indicator we're going to be focusing on here is the level of confidence on a four-point scale, I believe, of intermediaries, men and women, in their ability to mobilize communities to advance gender equality in target countries. Beautiful.
We understand that the effort, the resources, the trainings that have involved these women and men has been targeted at improving their capacity to mobilize their communities, you know, to advance gender equality. Beautiful. However, this particular indicator is suffering from self-reporting bias again. Here we're talking about perceived capacity. If you ask people, did your level of confidence increase after our training? They could say yes, but that's really subjective.
That's really subjective. And sadly, that might not be a valid measure of technical ability. It might not be. Right. David? No, I was just going to add, if you read the last part of the outcome indicator in their ability to mobilize communities to advance gender equality in target countries, ability to mobilize communities, you could also, as an alternative, just look in the future to see how many communities they mobilize.
And you could have some sort of crude definition of what do you mean by a community being mobilized? Maybe a couple of meetings, a criteria that they have to meet. That's another way you could measure whether they've actually done it. Exactly. Forget about asking them if they feel they have the ability. Go and find out. Yes, I agree with you, David.
Find out how many communities they actually mobilized and then compare that to another area in the country where the project is not operating and see how many communities they mobilized, and that would be a way of showing that communities are actually being mobilized. But they haven't done that here.
Another way of looking at this outcome indicator is to just look at whether these women and men actually mobilize these communities, and that would show that they do have the ability to mobilize, and there could be some minimum criteria, how many communities, and exactly what is a minimum criteria to reach to say, da, da, da, da, we've mobilized this community.
And then they could compare that with a group of men and women in another part of the country that are not even involved in the project and how they have mobilized their communities. And you would expect in this project to be a higher level. Or if you don't want to compare, just look at the number of communities that have been mobilized over time for the target men and women in the project. That would be another way of showing that our project has indeed succeeded.
Either way, but this asking how you feel is unacceptable. Yes, because that's exactly, that's so, so, so subjective. And David, I could have, my confidence level could be improved about something, but that does not actually translate to me actually doing it. Do you understand? Because there are other contextual and environmental and socio-economic factors that actually play into the actual decision to mobilize communities to advance gender equality.
So it's not just about, you know, we've done the training, we've trained 500 people, we expect them to do this because your confidence levels have improved. I don't think it's such a linear equation. Yeah. That's a good point.
And it raises an issue for those outcome harvesters out there, still trying to get one on the show to talk about the fact that in outcome harvesting, you could go to the end, see how many communities have been mobilized, and then work your way backwards to see if the mobilization of those communities was actually due to the project. Absolutely. And that's what I think.
I'm not an expert on this, but outcome harvesting, I think when I read the description of that method, that's what they try to do because they argue, you know, the whole world is complex and you can't do a linear connection, which is true. But there's not even an attempt here, as you can see. And some of the other projects we've talked about on the podcast, they do use most significant change outcome harvesting, but so far I haven't got them on the show. So I'm quite excited.
Hopefully someday we'll get somebody on the show that can explain it and actually show that in their performance measurement framework, we can show that our outputs are achieving the outcomes. But right now we're not there yet. Good point. Yeah, yeah, absolutely. Yeah. Go ahead. Yeah. The next one is increased efforts by select women's organizations to influence duty bearers in fulfilling their human rights obligations related to gender equality in target countries.
It's the same outcome I think we read before. So I'm talking about it again here and the outcome indicator is type of efforts made by select women's organizations to influence duty bearers in fulfilling their human rights obligations related to gender equality in target countries. And again, it's like type of opportunities I talked about earlier, type of efforts.
True. It's in the PMF, if you look at it, it says duty bearers who receive these efforts going from none up to frequent efforts is not good enough. That's what I point out here. Duty bearers are interviewed, which is good. They not the women organizations who report their frequency effort should be asked not how many times the women came and engaged with them, but something like this. Do you believe those efforts were a sufficient quantity and quality to advance gender equality?
So they should get right at it, go right to the chase, ask the duty bearers, we've got these women coming at you from this project, cost to the taxpayer of 18 million Canadian dollars. Have they done their homework? Do they look like they're ready? Do they know what they're talking about when they're advancing gender equality, when they engage with you? Forget about how many times they show up. What's the quality like?
The irony here is they interviewed the duty bearers in the performance measurement framework, but they asked them the wrong question. They should really probe, are you really doing a good job here? And they don't. Now, to be fair to them, maybe it's buried in their measurement tool somewhere, but I don't see it in the performance measurement framework. All we see is the scale, none up to frequent efforts. And that's clearly not a good measure of their ability to influence these duty bearers.
Yes, yes. We're supposed to be checking for the effectiveness, their effectiveness in influencing the duty bearers, not necessarily the frequency. And the irony is they're getting there. They've done what we've recommended in earlier indicators. They've said, forget the women, we're going to go straight to the duty bearers because that's where the impact is supposed to happen. We're going to ask them how well they're doing.
But in this question, they just asked them how many times they knocked on the door. So yeah, it's ironic because they're kind of in the right direction, but they're not asking the obvious question, you know? So they're getting there. Next outcome. I, sorry, going back to, sorry, David.
That's why it would have really been interesting to have a representative from Equitas here because maybe they could have provided a bit more exposure to their rationale behind just stating things like this, because it's possible in your brainstorming sessions, they might have thought, oh, we can't do this, but we can do this. Maybe they've thought about what we've just said, but canceled it out. And then they just put this in. Do you understand?
It would have been really exciting to hear from them to know why they stated this indicators as they are. Yeah, absolutely. But go ahead. Yes, I agree. And I think I've alluded it to in my trailer. Yeah, I think I have, which is one of the problems is Global Affairs Canada requires this performance measurement framework before they get any funding, but they're not following their own guidelines. They could say, look, these outcome indicators are not valid measures of your outcomes.
What's going on here? And that's one thing that is an issue here. And they may have another document buried somewhere that explains why they went ahead with this outcome indicator. But based on the performance measurement framework, it's still an issue, but you're right. It would be good to have a representative on to provide more detail about why they pick these outcome indicators. And sometimes it's an afterthought just to get the funding. I don't know.
That may be an unfair statement, but that's the way it's happening these days. So I think we're on enhanced collaboration again among women and human rights organizations to advance human rights in their countries, and you have an outcome indicator for there. Yes, and the indicator in this case is the degree of collaboration on a four-point scale among women and human rights organizations to advance human rights in their countries. Beautiful.
However, this still points towards a self-reporting bias because they need to measure externally and not ask the organization to report the frequency of their collaborations. We need to also measure more than just baseline and endline. So I feel that this is a good indicator. I don't think it's bad in and of itself, but however, the measurements for this indicator should have a broader scope than what they presented.
Yes, and most important, well, I think importantly, again, they shouldn't be asking the women's and human rights organizations to report on their own efforts on how well they are collaborating. It's just a huge bias.
They should have an external third-party monitoring going in and going in and using their own criteria figuring out if over time, here we've got just a baseline and an endline, are they actually increasing their degree of collaboration between themselves because it is an important activity, for lack of a better word, or outcome that you want to increase collaboration to get this critical mass so that you can advance human rights in the country. But don't ask them.
Go out and do it independently and show that the project is actually in some way through networking or whatever, because there's a whole list. If you read part one of this podcast, describes all the, there's 10 target groups, all the activities that they're doing is in terms of outputs to achieve these outcomes. So yes, it has to be an external measure because it just defeats the whole purpose if they just ask themselves. Yes. Right. Yes, I agree with you.
Okay. So the next one is increased leadership of women and men, community mobilization activities that advance gender equality in target countries. And the outcome indicator for that outcome is, quote, percent of total women and men trained demonstrating leadership in community mobilization activities that advance gender equality in target countries. This is just attendance. Everybody goes home. They say trained demonstrating leadership.
So that implies that during the training, they demonstrated leadership. So how do they measure leadership skill? It's not clear in the performance measurement framework. So it looks like it's just attendance. I don't see in the performance measurement framework, again, the following measurement tool was measured, used to show that X percent actually showed leadership after the training that they didn't have before in terms of leadership skill, for example.
Yeah. I don't see that in the performance measurement framework. Even if we did, and I missed it, they're not using a comparison group. So we can't show that the training was directly responsible for an increase in leadership skills. So that's the problem there. They only measure baseline and end line.
That suggests only at the beginning of the project and at the end of the project, which means in the middle, they could easily, even if they have a proper measuring instrument, they could easily teach themselves these leadership skills that have nothing to do with the training on leadership. See what I'm getting at?
Okay. Yeah. So even if they have a measurement tool, I can't find it in the PMF, but even if they have a measurement tool that measures their level of demonstrating leadership and it's gone up and the percents are going up, they only do it twice, once at the beginning of the project and once at the end, which it's a four-year project.
So there's just no way they can claim that the training has increased the leadership skill levels, even if they claim the percent of total women and men trained demonstrating leadership has gone up from the baseline to the end line, right? Just two measures is not good enough. You have to measure frequently along the scale to show that the project training is responsible.
Like I say in my trailer, after the first training, it could go down the road and figure out how to do this leadership skills on their own with somebody else. And then they come back four years later, hello, percent goes up. Yeah. They have nothing to do with the project training.
Yeah, but David, in a little bit to their defense, in the PMF, they specified for specific countries and now they're taking the example of Tanzania, they said they listed four specific activities, awareness raising campaigns, capacity building activities, engagement with duty bearers, support to service providers working with victims of gender-based violence.
So I guess maybe they're trying to check amongst this cohort of people, of men and women that we trained at this specific point in time, how many of them from each cohort led awareness raising campaign programs for Veterans warmth day leader and community. That's a little off the top of my head.