Welcome to the Improving Development Evaluation Podcast. I'm your host David Wand and welcome to episode six, part two, where we continue our discussion on colleges and institutes Canada and their $18 million project funded by the Canadian taxpayer that is operating in Senegal and is entitled, A Thousand Women, I Am Woman, I Exist, I Participate.
And just to give you a brief recap, I contacted the colleges and institutes Canada on March 9th to invite them to this podcast to respond to our critique of their performance measurement framework. They did not reply. I then followed up on March 18th, along with a phone call and left a message. We still have not received a reply. Today is now March 28th and they have decided obviously not to attend this part two of the podcast.
But fortunately, I do have with me an evaluation expert that I will introduce to you shortly. This $18 million project, looking at the performance measurement framework, had nine outcomes and used 17 outcome indicators to measure whether or not those outcomes had been achieved for the project.
And I led to the conclusion that colleges and institutes Canada could not make the claim that its project was achieving its outcomes simply because the outcome indicators were not properly measuring the achievement of the outcomes with only one out of the 17 outcome indicators properly measuring whether the outcome was achieved.
So with that being said, I'd like to introduce to you Dr. Jenny Yau Jorgensen, holds a PhD in system safety from Lund University and she is in Sweden and has been kind enough to participate in part two of this podcast. And she is also an evaluation practitioner. Welcome to the podcast, Jenny. Thank you for having me, David. Good, great. So what we're going to do is we're gonna go through each of the nine outcomes and select some indicators.
And we're going to in detail, describe why they are flawed in properly measuring the outcome. And we will start with a few outcomes that are actually in the performance measurement framework as outputs, but we put them as outcomes because they expect outcomes to be achieved. So starting with the first one, the outcome was quote leadership training led by male leaders in the communities. And you would expect something to be learned from that training.
And the outcome indicator that they stated was and I quote, number of people benefiting from leadership training by male leaders. And the problem with that indicator is that it's just assuming that people show up, they take attendance, number of people, but we don't know if they actually benefited from the leadership training. It's just taking attendance.
And we would prefer obviously, that they would go a step further and actually measure whether or not they learned the content of the training with respect to leadership. So moving on, I will now state the second outcome and then Jenny, you can state the outcome indicator and make your comments on why you believe it may or may not be properly measuring the outcome. The outcome is and I quote, strengthening workshops on leadership organized with women and men leaders.
And what would be the outcome indicator that you're looking at? Right, then the indicator that we analyze is the number of men and women leaders strengthened in leadership. So the issue that's sort of associated with this indicator can be looked at from different perspective. One is that strengthening, strengthening leadership is such a broad term. So that means the indicator itself would require different kinds of qualitative indicators.
At the moment, the focus is mainly on the quantitative, the number of leaders being strengthened in leadership. So that's one aspect we thought the indicator could be enhanced. That's great. Okay, thank you, Jenny. And we'll move on to the next outcome. Consultation mechanism with women's groups, economic interest groups and organizations. And the outcome indicator that you're looking at? Is the number of consultation mechanisms established and functional.
So again, the issues with the functionality, which also require indicators that goes beyond the quantitative. What kind of mechanism? Is it about addressing the effectiveness and different qualitative sort of matrix, for example? Is it really like regularity of meetings? Are we talking about that? Or is it talking about decision-making processes? And what are the follow-up actions? So these are sort of missing, missing matrix in this over quantitative indicator.
Right, and I'm doing a similar one, the mechanism for consultation between government partners put in place integrating gender. And the outcome indicator there is number of consultation mechanisms established and functional. And the key thing I would add to what you've said is, they've written right in the outcome indicator functional.
So you would think they would say, okay, let's get a tool, measurement tool, as how do we define and measure that that consultation mechanism is actually functional? And they don't. All they do is count the number of mechanisms they've produced on a piece of paper and maybe used, but they haven't gone the next step, which is what's the expected outcome of that mechanism? It is, it's functional. So what do you mean by it's working? It's functional. How are you gonna measure that?
And that's missing in this performance measurement framework. So they can't claim it's functional because they're not even measuring it. Exactly. Maybe just to add a little bit, going back to my sort of similar indicator, because there's actually, in the outcome statement, is looking into the economic interest groups and organizations. So in terms of measuring the functionality, I think another aspect kind of missing is about the inclusivity and accessibility by underrepresented groups.
So that is quite missing out in the indicators, the way it's being formulated at the moment. Right.
And as we get further down, we'll return to your point about decision-making, because in particular with women in the project, and I should elaborate and recap that in part one, major part of these services of the project is for the vulnerable women to receive training in entrepreneurship and leadership, to the point where they have more power in decision-making bodies and they have more money from learning how to run a business. And that's basically what they call the empowerment.
So we'll get to that. So that's a good point about the decision-making that you raised. So the next outcome is increased community awareness of the importance of women's participation in decision-making, and the outcome indicator you'll be looking at to measure that outcome? Yes, it's the number of people in communities directly sensitized to the importance of women's participation in decision-making. Desegregated by sex and occupation in practice leaders or not.
So the issue that we find about the indicators that while the indicators track the reach, the outreach of the sensitization efforts, what is missing in terms is kind of related to the quality or the depth of the impact, for example, the attitudes, whether attitudes change after the sensitization. So that sort of aspect were not really captured in the indicator.
Another bit is about, you know, in terms of who actually came forward to this, who are being reached out in terms of these activities, those were also not quite captured. So some other qualitative measures might be needed. Right, and I would add to that also the fact that they do this from the performance measurement framework.
You can see it's an annual survey with targets to reach levels of awareness, but how do we know these targets were reached due to the project services of raising awareness? We don't because they only measure it once and they didn't use a comparison group, even though interestingly enough later on in this podcast, they actually do use a comparison group.
So it begs the question, why did they not use a comparison group here to see if communities outside of where the project is operating also got sensitized, but at a lower level? And so that's not being done here. And also, did they ask the people surveyed if they knew about the project and if no, use them as a comparison group? So that was a key thing I would add to what you've said, yeah.
Yeah, and then we have quite a number we will get back when it comes to other indicators, it's about the desegregation. I think there's more intersectionality aspects we can look at, but we will come back to the desegregation issue. Sure, yeah. The next outcome is increased capacity of men and women leaders to intervene in communities. That's the outcome.
The outcome indicator that I'll be looking at is, quote, perception of community members on the effectiveness of the promotion of gender equality by the project's gender equality chance and gender equality champions. Now, the issue here is it's measured only once per year through focus groups. What about other communities that did not receive the leadership training? How are those women doing on leadership there?
So we have this issue of capacity of men and women leaders to intervene, but they could be intervening in areas where the project is not operating. And so again, it's only measured once a year, and it could be very easy that they could have figured this out all on their own without the project services teaching them how to intervene in these communities.
And even the perception of community members on the effectiveness of promotion of gender equality could be going up in these other communities where the project's not operating. So it's the same problem of where they're not measuring. If they don't want to use a comparison group, that's fine. But the other issue is if they're only measuring once a year, that's not enough.
They need to measure frequently before they start the project, during, during, during the project, and after to show that there's a trend where the sensitization is going up. Yeah, otherwise there would be causality issue, right? How do we claim that is the contribution of the project intervention? Right, and you don't necessarily need a comparison group. You can just do proper quasi-experimental and measure several times rather than just once. You're exactly right. You can't make these claims.
You can at least try to improve on it, that the perception levels are moving in the right direction. But if you want to claim that, then you've got to measure more frequently, right? Yeah, over time, tracking the over time, yeah. Exactly. Thank you. The next outcome is increased influence of women as active citizens in their communities. And you're looking at an outcome indicator for that outcome. Yeah, we have a couple of indicators here.
So the first one is the percentage of women targeted by the project who participate in decision-making bodies in their community. So this indicator measures the participation rate of women, again, quantitative in decision-making bodies, which is a good start. But it really doesn't reflect or capture the level of influence on the effectiveness of their participation. So again, this participation does not always equate to having an active role or voices.
We have seen even from our experiences, sometimes the typical gender role, taking notes, are happening in many of these decision-making bodies. So it's important to look at what kind of roles, what kind of influence do they have. Can they set the agenda, for example? And does it really make a difference if those decisions are being initiated by women? So these are some missing aspects. Yes, and I would add to that, again, it's only measured once a year.
But it may not be a problem if it's only once a year, as long as they look at the decision-making bodies, like you said, and see what exactly do they have any power in making decisions. And has their representation as women gone up over time? And we just don't know. We just know the percent of women targeted, but they don't go into detail about. It's another issue of defining and measuring actual influence. And just being a member of the decision-making body is not enough, is what you point to.
Absolutely, yeah. So we'll come back to some of the other two indicators later on. Right. And I think we're going to the next one, which is increased inclusion of women as actors in social and economic development. And you have an indicator there that you want to look at. Yes. So that's the percentage of women targeted by the project who have improved the management of their income-generating activity. So this, again, is what do we mean by improved management?
That's a broad term that requires a specific matrix. But again, it's also about why not measure the actual improved income generated because of their participation, their empowerment through the project. So these are kind of the missing aspects in the indicator. Absolutely, yeah. And I've noticed this in another episode I did, where they just seem to get scared of using the I-word, income. They will talk about savings plans, but they won't say, show me the money.
And this is a good example where if you just show me the money, please, and this will show that women are actually inclusive, being included in economic development because their incomes are rising. And you would think it would just look at that. Exactly. Again, you can categorize it almost as an output rather than an outcome indicator. Yes, yes, exactly. Because again, they're not measuring increased. And the key word is increased inclusion.
It just says percent of women targeted who have improved the management. So maybe their management has improved, but that's evading the actual inclusion of economics. Exactly, exactly. And then the last outcome of the nine is strengthening the empowerment of women among the most vulnerable across Senegal.
And this is where the irony comes because this is the one indicator at the very top of the food chain, results chain, where they are actually using what's called the Female Empowerment Index, FEMI. And they probably named it after the well-known musician from Nigeria, which is kind of cool. Give them credit for that, even though it's Senegal. And it's good. They've used a comparison group.
So they've actually set in the performance measurement framework, we're going to use a group of women that are not participating in the project. And we're going to measure their empowerment levels on this index. But the problem is it's a bit late. So even if they can show, the first thing they need to do is even before the project starts, hopefully they've done this, they've shown that there's no statistical difference between the two groups to begin with.
That's how you start before the project. And then you can show over time that the group that they are engaging with in the project is actually their index is going up in a much greater rate than the control group. And that could be happening. We don't know. Because the other issue is we don't have the data. They only provide a blank PMF. And if you go to the project browser, Global Affairs Canada, they don't give the data for this indicator, which is ironically one of the few good ones.
And there's no data available. So we don't know. Yeah. Sorry, go ahead. I quite like what they use in the indicator, like in brackets, adaptive version. That means they are quite sensitive to adapt to the local context. But in terms of improving the indicator, they could also measure a bit of what we call it the process indicator.
It's like to what extent that the adaptation process to the local context is addressing the cultural diversity sensitivity, for example, to make those indicators relevant. So I feel that that could be an area for the organization to rethink about these kind of composite indicators. Yes. And to add to that, there are six components of the FEMI index, the Female Empowerment Index. And two of them are relevant to this project. One is employment, achievements, and achievements on decision making.
And those two, they obviously I think have adapted so that they're going to be asking the women specifically on how are you doing on employment and how are you doing on decision making. But the irony is, as we've just covered, is that for employment, they're not measuring incomes in the project. So even if they say yes, we have no data to support their claim because it's biased, because they're self-reporting it. To what extent the income has actually increased. Right.
They could say yes, but we just don't know. Because the indicators underneath the FEMI empowerment, like employment, they're not measuring income. And on decision making, as we just discussed, which is the other dimension of the six they could use in the index, they're not looking at, did they increase their power on these decision making bodies? We just don't know.
So that's the irony in all of this is they've got this wonderful indicator at the end where they can wave the flag and claim that their project women compared to the comparison group have become more empowered.
But then when you ask them, what do you mean by more empowered, you go to the performance measurement framework and there's no support for whether or not their incomes went up or whether or not their decision making power went up because they're not asking them in the performance measurement framework for those indicators. So that's what's really interesting about it is it's kind of like a smoke and mirrors, as they say. But that's why we're here is to bring that out.
And if they were to improve the indicators the way we have discussed them, then they could make the claim that the reason their empowerment has gone up, or assuming it is, we don't have the data, but if we get the data, it would show that. And it's because we can show their incomes went up compared to the comparison group, their decision making power went up compared to the comparison group, et cetera. So we just don't know, but at least they're trying to move in the right direction.
So what we can do now is go back to the outcomes and look at the other indicators where we could cover all 17 because I think we still have a bit of time. So we'll go back to the outcome strengthening workshops on leadership organized with women and men leaders. The other outcome indicator for that was number of leadership workshops held. And it clearly falls into that problem category of just attendance and everybody goes home, as I say in my trailer.
They don't even measure whether or not they've strengthened their leadership skills or knowledge with respect to leadership after the trading. So that's a problem there. The next indicator was for, sorry, you wanted to make a comment? Go ahead. Yeah, I would just like to add to in terms of, they could also track in terms of the active engagement of both women and men participants, but maybe even organizers because those are the women and men leaders who will organize this training.
So in terms of the engagement, what kind of roles they are also playing out in running these workshops, that could also be an interesting aspects in setting the indicator. Oh, yes, because one of the target groups is male champions. And that's a key, very important point is disaggregation and seeing if the men are actually championing and moving in the right direction in that area.
Because right now all we have is we're assuming that their leadership has been strengthened from the training, but again, there's really no measurement. The pre-post, right? The pre-post workshop assessment is missing out. Yeah, even a pre-post is limited, but at least it's better than nothing. You could do in the same day, figure out if they've actually learned and their levels have gone up, even for that just workshop on training with respect to leadership, yeah.
So the next one is increased community awareness of the importance of women's participation. And what I was looking at there is the other indicator was, quote, direct beneficiaries in the project through focus groups also measured on importance of women's participation in decision-making. Well, of course there's self-reporting bias there. They're holding a focus group. It's of the people that are participating in the project, the women.
So they're obviously a bias for them to say, yes, I've become increased aware of the importance of women's participation in decision-making.
So this is a case where I believe focus groups are improperly being used, and this is a common problem, where instead of using the focus group to compliment a problem on an indicator, to say, why are you not achieving your target on this indicator, going in and having a focus group and asking them, which is why you use focus groups, to compliment a quantitative indicator. They're instead using it to replace the quantitative indicator measure on community awareness, for example.
So this is a problem that I often see. Yeah. Totally agree, yeah. The next one's for you. I'll read the outcome. Next one, yes. Increased capacity of men and women leaders to intervene in communities, and you have an outcome indicator for that. Yes, the number of men and women trained in leadership who carry out gender equality sensitization activities in communities desegregated by gender.
This indicator measures the quantity of activities and people trained, yeah, but really does not necessarily reflect the quality of public awareness activities and what came out of it. So what they could actually improve is to really maybe have a follow-up study to actually see does these kind of activities generate some sort of behavioral change or attitude change in the communities.
But again, I think that desegregation by gender is good to have, but depending on the communities, you might need also other sort of different matrix to understand the social economic background of the community members who actually come forward to participate in these activities.
Yes, it seems quite complicated, like when you look at it, how are we gonna say that these men and women who showed up for these project trainings actually increase their capacity to quote intervene in communities effectively? I mean, the first part is do they have the technical capacity to go into these communities? And that's the immediate outcome that they put here.
But you've made a good point about even if they get to that level, what's gonna happen next is actually gonna lead to the awareness and changes at the, downstream as they say. So yeah, attendance simply is just not good enough. Yeah, absolutely. And the next one you're gonna be doing is for the outcome, increased influence of women as active citizens in their communities.
Yeah, and the indicator that I was looking at is the perception of members of community organizations, men and women on women's participation in decision making within communities desegregated by sex. So this is again reliance on the subjective perceptions which have already we discussed earlier on, which can be influenced by personal biases, cultural norms.
So we might need other sort of qualitative approaches or other sources of data to substantiate or at least to unpack some of these perceptions, subjective perceptions. Yeah, and then another aspect just before going back to you is again, the desegregation just by sex. I think it also kind of overlooking on other social economic dimensions, for example, age, ethnicity, disability, social economic status.
So I think that desegregation is good, is a good start by gender, but a lot of indicators, you know, kind of miss out on collecting data that desegregate on these other important dimensions. Right, and it's also only measured once a year. So it's very difficult for the project to make the claim that their trainings with these women have actually led to the achievement of the outcome, which remember is quote, increased influence of women as active citizens in their communities.
And the way they're gonna try to show that is by asking, I think it makes sense. They wanna go to the people on the other end, the communities that have received this engagement from these women and say, hey, what do you think? Do you think these women actually, you know, influenced you in the desired direction? Women's participation in decision-making in communities. That's basically what they're looking at in terms of the increased influence.
But it seems to be quite a challenge to actually show that. But only measuring once a year is not good enough. They have to do it more frequently or also, or pick a comparison group and ask another group of women outside of the project, sorry, other communities, what they think of women's participation in decision-making and show that there's a major difference or measure more than just once a year. Exactly. The next one. Next one is also mine, right? Yeah, I'll read the outcome.
Increased influence of women as active citizens in their communities. That's the same outcome. But I think the outcome indicator is slightly different. Yes. Yes, this is about the perception of women targeted by the project on the participation in decision-making within communities and in the family.
So, but again, it's kind of similar to what we talked about previously is about the perception, again, that can be sort of affected by other sort of internalized gender roles, self-esteem and societal expectations. So this may not align with the actual.
So instead of measuring the actual, which we highly recommend in terms of the influence of their participation, purely relying on perception may not be sufficient to really be an indicator for the outcomes, the increased influence of women in the, as active citizens in their communities. And in this case, it's in the family.
So, you know, what we could sort of an improved indicator would could, for example, is maybe, you know, getting some information from the family members, like community members, for example. So this again, self-reporting bias, I think David, you already talk about on some other indicators that also comes into this indicator. Yeah, exactly.
We preferred that they use the other approach that they used in the previous indicator, go to community members, not the women themselves in the project, or like you said, quite rightly, other family members, because I think they're claiming here that their decision-making power is going to go up even within the family for these women, not just in the community at large.
And this is in other episodes, sorry to bore you, Jenny, I'll bring it in, in Vietnam with Care Canada, we raised this same problem where, how do you get at measuring increased women's power in the household? It's quite tricky, but you could go and ask other members in the household, but that's not even done.
Self-efficacy is important, yeah, but at the same time, I think for, in order to generate robust evidence base, other type of data collection methods would also be needed to strengthen the evidence base. Right, exactly. The same outcome, increased inclusion of women as actors in social and economic development, sorry, that's not the same outcome, it's the next one.
There was another indicator they used called, and I quote, percent of women targeted by the project who are business leaders or entrepreneurs, that it just doesn't go far enough. They could tick off the box, but are they earning any money? And are they earning more money than those in the, that are not participating in the project and not receiving business training, leadership training, et cetera?
So you definitely need to have a comparison group, or even without a comparison group, it's the same problem. They can't just tick off the box and say, I'm a business leader and raise their hand. They've got to show that they've actually earned income as a better measure of increased inclusion as part of economic development. And the next one, indicator for that same outcome, increased inclusion of women as actors in social and economic development.
Another indicator they used in the performance measurement framework was percent of women in employment or self-employment after training.
So again, the problem there is the same, is it's a good idea that they've been employed, but they need to show that the training from the project has made the higher percentage of these women become employed or self-employed, specifically with income, compared to women who didn't benefit from the project, didn't receive the services such as leadership training and entrepreneurship. Maybe another aspect is also the nature of the jobs, the employment, is it related to the training or not? Exactly.
Disaggregation, please. Absolutely, yeah. And I think we've got one more here, increased inclusion of women as actors in social and economic development. And again, one of the indicators is perception of women targeted by the project and men guarding their socioeconomic situation. So your point's a good one, Jenny, that they're disaggregating the men and the women, which is good, to see if both groups are moving in the right direction. The problem is, is it's the targeted by the project.
So again, you're gonna have some self-reporting bias where they're gonna say, oh yeah, my measure, or however they define it, of being included in the economy has increased. But they need to use other measures that we've suggested before to get at this increased inclusion rather than asking the project participants themselves because of this self-reporting bias. Yeah, you also have an earlier point about the targeted group. Are they reaching out to the targeted group?
Because it's all about the underrepresented or marginalized women and men. But again, it's that, are they, are the target group, have improved their socioeconomic status or situation? That is also of interest to measure the outcome. Oh, absolutely, that's one of the issues is, I've found with many projects, the vulnerable women as they identify them here, sometimes they're not even reached. Exactly.
And it's the women who run the NGOs who benefit the most from the project in terms of salary, et cetera, and the actual delivery to the vulnerable women is minimal. And in this case, the measurement is certainly minimal. So it's very difficult to reach.
So I think we can conclude that, at least from my point of view, that this project in its current design from performance measurement is not adequate to support the project in colleges and institutes Canada claiming that they're achieving their outcomes. That is, that women are becoming more empowered economically and politically. But that's my opinion. What do you think, Jenny?
Now, I agree with you with your conclusion, but I also want to applaud the organization for setting actually very good outcome statements. But I think there's really a space as we, the objective of our podcast is actually try to improve practice when it comes to formulating the indicators. So if there's already a lot of investment in setting out very good outcomes, with suggestion then why not just also improve the indicators?
So there is actually a very clear, more clarity and specific matrix that actually generate evidence base to support the argument, whether the outcomes have been achieved or not. Well said. Of course, I have a bias myself, setting up this podcast for that whole intent and purpose. So thank you, Jenny. Thank you so much, David. Yeah. Thank you for having me. You're welcome.
And so what we'll do now is the last sentence in this podcast or a few summary remarks is that, we will send these part one and part two, two colleges and institutes Canada, but more importantly, we'll send it to the honorable minister for international development, as well as the shadow critics for international development in the conservative party, the new democratic party, the Bloc Québécois party, as well as I think maybe even the leader of the Green Party.
And we'll leave it at that and continue to recommend that one, they make the performance measurement frameworks available to the public on the Global Affairs Canada website. Number two, they actually provide the data from the indicators. Here, we don't even have the data. It's just a blank performance measurement framework. And also finally, as we've discussed, the Global Affairs Canada should follow their own guidelines.
And that is when they approve a performance measurement framework, they should be saying, why are your outcome indicators not properly measuring your outcomes? Please revise before we disperse $18 million. Finally, if you're gonna be a charity, become a charity and don't make these claims of these outcomes that you're achieving. There's nothing wrong with that, nothing to be ashamed of. And that's the other option they can have, these organizations, is don't have any outcomes at all.
Just deliver the services because everybody needs them and everybody goes home. There's nothing wrong with that as another option. What do you think, Jenny? No, that's a very good remarks to sum up. But as a practitioner, I also suspect that there's actually maybe updated performance measurement framework that we're not really shared. So maybe there are some ongoing substantive work to improve the monitoring evaluation, but maybe these kind of documents are not always shared externally.
So I also encourage implementing agency, particularly with such a huge budget, to really make this kind of information also available. That's a good point. And also some of these NGOs have claimed they don't have the evaluation expertise to design proper indicators or they don't have the resources or money to have proper comparison groups because it's expensive, you have to measure another control group. Or as we've advocated, you have to measure more frequently, which costs more money.
But my position is the government of Canada should say, here's your 18 million, but by the way, 10 to 15% of it has to go to monitoring and evaluation. Right there. I think what, yeah, I totally agree with you, David.
There's always the budget concern, the expertise concern, but even though there's a wide range of resources out there to improve monitoring evaluation practices, the issue is I think what we're advocating is that maybe steer a little bit away from the desire to just quantitative indicators could maybe perhaps use a mixture of quantitative and qualitative indicators. So at least you can have more substantive, more robust sort of measurement to measure results. Yeah, I agree.
As I mentioned earlier, some people may suspect I have a quantitative bias, even if that's the case, they have to be complimentary. You can have increased inclusion. You can have a quantitative indicator for that, but you can also have a focus group qualitative to compliment that by saying, okay, we have statistical significance here.
It's clearly shown that your incomes have gone up greater than the control group, but we'd also like to know what worked, what didn't in terms of increasing your incomes. And that could compliment the actual achievement in the right direction of the quantitative indicator. And you certainly can use both. The problem is they often don't, and they often miss the mark on the quantitative as we've I think highlighted here, or they abuse the qualitative by using it just by itself.
So I think we're moving in the right direction, yes. Yeah, maybe just one last comment, if I may, David. Of course, we don't know the actual implementation of the monitoring evaluation or performance management framework, as you call it, that we also encourage, at least from my point of view, to encourage, if that wasn't already done, collaboration with the local research institutes to collect this kind of qualitative data or quantitative data, so to improve actual measurement of outcomes.
Absolutely, and in fact, I think a lot of the organizations do that, use local research institutes. I've been involved in that when I was in Nigeria doing third-party oversight on local Nigerian survey firms that were collecting the data for the indicators. Absolutely, yeah, that's a very good point. In my trailer, I talk about that's important, but what's really important is the initial design, where if you don't design the indicators properly, you have garbage in, garbage out.
And that's the focal point of my podcast, but yes, absolutely, you have to use local firms, and it's quite fascinating how many local firms in this global economy have expanded by relocating in developing countries. There's quite a few very impressive local data collection firms that have developed to address exactly what you've raised, yes. Yeah, maybe on that note, because I remember in your part one podcast, you talk about one of the target group of the project is the project staff itself.
So maybe that's also an avenue, if that is not already done, is to include performance management or measurement capacity as part of the key competencies to develop the local project team members, sort of part of their competence development framework. Oh, absolutely, and that's a big issue, is some of these NGOs will respond that they don't have the technical capacity to do proper measurement and evaluation, and it's always a challenge. Absolutely, yes, very good point.
So I think we'll end it at that, and we'll send it off to the minister, and hopefully we'll build this slow body of evidence where maybe someday performance measurement frameworks will be on Government of Canada websites for the four to five billion dollars worth of projects that they have every year, and the indicators will properly measure their outcomes. Okay. Thank you so much, David. Okay, we'll keep in touch, and maybe you'll be on another podcast if you want. Thank you.