Welcome to the Improving Development Evaluation Podcast. I'm your host David Wand and welcome to episode one, part two, where we feature Care Canada as our international development organization. Before I introduce you to our two evaluation experts, I'm gonna give you a brief synopsis of what I've done with part one. On October 19th, I sent to Care Canada by email part one of this episode, explaining my critique of their performance measurement framework.
And then on October 20th, I followed up with a phone call to make sure they had received the email. And then I followed up again on October 30th with a repeat email reminding them of the invitation to attend the podcast. It's now November 6th. We have not received a reply from Care Canada regarding whether to attend and we have not received a written response, which was another option for them.
If they were unable to attend, they could send a written response to the critique of their performance measurement framework. To be fair to Care Canada, today it's 7 a.m. Ottawa time, where Care Canada is located. That's a bit early in the morning, but they could have sent a written response, but we haven't heard anything from them. So now we're gonna proceed and go into more detail about the outcome indicators and why they're flawed in their performance measurement framework.
From Georgetown, Guyana, I have Donald Cole. Donald Cole holds a master's of science degree in data engineering from Edinburgh Napier University and is a monitoring and evaluation specialist. And we also have from Kampala, Uganda, Dr. Mark Degner, who holds PhD in social and economic science from Karl Franzens University in Austria. And he is currently a technical advisor in advocacy, communication, and documentation. So what we're gonna do now is we're gonna go through the outcomes.
So let's start, I'll read out the outcome, and then Donald, you can talk about the outcome indicator and why it's flawed, and even if you wish to provide an alternative indicator. So beginning with the first outcome in the project, increased entrepreneurship capacity of women-led grassroots social enterprises to pursue economic opportunities in Vietnam. That's the outcome and the outcome indicator?
Yes, the outcome indicator for that particular outcome is number of women-led grassroots social enterprises reported increased capacity to engage in entrepreneurship opportunities in Vietnam. So this particular indicator, there's like three main points I would like to highlight for this indicator. The first one is the vagueness in increased capacity. The term increased capacity is vague and it does not really specify what aspects of capacity are being measured.
The second point is its reliance on self-reporting. And that data can be very biased because self-reporting data persons tend to be very biased in terms of how they report and provide feedback. And the third key point on this indicator is the lack of objective measures. There are no objective measures for entrepreneurship capacity or success.
In terms of improving this particular indicator, the first thing would need to be done is to have a clear definition, obviously, of what capacity entails, whether it entails business skills, access to capital, market access, et cetera. The second thing that would be needed to address is the objective matrix of the indicator. What are the objective matrix are we going to measure? Are we measuring revenue growth, employment rates, number of new businesses started, and so forth.
The third thing that would need to be addressed is periodic and independent assessments. Conduct periodic assessments. And these assessments should be done by independent evaluators to ensure that they measure the growth and the sustainability of the enterprises. And then there's an issue with confidence versus competence. While confidence is important, it should be measured alongside competence, especially when it comes to entrepreneurship skills.
That would be my brief critique of this particular indicator and ways in which this indicator can be improved. I didn't come up with a rewarded indicator because I think I would probably need a bit more information, but those key points would definitely address the issues with this particular indicator. That's excellent, Donald. Thank you. So we'll move on to the next outcome statement in the performance measurement framework for this project by Care Canada delivered in Vietnam.
That's increased gender responsive capacity of poor, rural, and ethnic minorities, especially women, to save, borrow money, to support productive and sustainable economic activities in Vietnam. The indicator I'll be looking at there is percent and number targeted poor, rural, and ethnic minority women and men who report an improvement in capacity to save money. And I have the same issue that Donald's explained is increased gender responsive capacity to save and borrow money.
They're depending here again on self-reporting. So you have this self-reporting bias or what some people call social desirability bias where they will have a tendency to want to say, yes, we've increased our capacity. And another point is, if they have the capacity to save money and borrow money, why not just check out their bank accounts? Why not just go right to the chase?
Some people may say that's intrusive, but if they really wanna show that they've been able to save and borrow money, they should look at their savings accounts. Their ability to borrow implies the ability to pay back a loan on time. So they could also use that as an alternate way of measuring their capacity.
And also as we allude to in the trailer of this podcast, even if their capacity is properly measured, and it is not here, but let's assume it is, how do we know it's the project training that was responsible for this increase in gender responsive capacity to save and borrow money? There could be another group of women in another part of Vietnam who are also doing this just fine, and there's no project training going on. They figured it out all by themselves.
So that's what my points would be for that. Now, the next indicator is 1210.1. So that would be Donald again. This is where we have the outcome, increased gender responsive capacity of poor rural and ethnic minorities, especially women, to earn income from climate smart and sustainable on-farm and off-farm livelihoods in Vietnam. Donald? Sure, thanks, David. This particular indicator that supports that outcome is percent and normal.
Targeted poor rural and ethnic minority women and men who report an improvement in their capacity to earn income. And this indicator is disaggregated by sex, ethnicity, age, disability status. Three main issues again with this indicator. The first one, again, same issue like the last two indicators. It's based on self-reported improvements, which may not reflect actual income changes. The second issue I have with this indicator is the frequency of measurement.
It's unclear how often this measurement is taken, which can affect the reliability of the data being collected. And then there's no comparison group. It's difficult to attribute any improvements to the project, specifically if you don't have a comparison group. How this indicator would need to be improved to measure the impact project is, the first thing that would be needed to address is, you know, having that control group being selected.
And that control group should be matched on key demographic and soco-economic variables to ensure compatibility and implements physical techniques such as propensity score matching can be used, for example, to select a control group that mirrors the intervention group as closely as possible. The second thing is the frequency of the data. Determine an optimal data collection frequency that balances the need for the timely data with the logistical constraints of data collection.
This may involve seasonal assessment to account for agricultural or market cycles that affect income. And then there's income measurement. A comprehensive income measurement tool would need to capture not only the amount of income, but also the sources and stability of that income. And this tool should be sensitive enough to detect changes over time and attributable to specific project interventions.
And the fourth thing I would suggest that would need to be considered for this indicator for improvement is link the specific activities to income improvement, to specify project activities, to identify what exactly is really working. So the next outcome is increased willingness of poor rural and ethnic minority women and men, duty bearers in particular, commune officers, to address the issue of women's unequal caregiving burden and economic rights in Vietnam.
And I think Donald, you're looking at the indicator there. Yes, that particular indicator states average score of ethnic minority men and women, duty bearers in particular, commune officers who report gender equitably attitudes. And this is disaggregated by sex, ethnicity, age, disability status, type of participant. I have three main issues with this particular indicator. The first one is the subjectivity of it. Discard indicators relies again on self-reported measures of attitudes.
And this can be highly, and in more cases are very highly subjective and influenced by social desirability biases. The second issue I have with this indicator is the lack of baseline and follow-up. Without the baseline measure or regular follow-up, it's very difficult to assess whether any changes in attitudes are due to the project or the activities coming out of the project. And there's no actionable outcome.
Porting an attitude does not necessarily translate into actionable change or improvement in gender equality. Let me just ask you, Donald, that's correct. But in this outcome, to be fair to Care Canada, they just wanna make sure that the attitude has changed in the right direction, increased willingness. So you mentioned lack of frequency in measuring this attitude change. If we have a chance, we can look at the performance measurement framework.
What exactly is the frequency they measure this attitude change? Does it say in the performance measurement framework? I think they mention outcome harvesting and it might be annual reporting. Let's check to make sure. I'm quite sure the frequency is not enough, but here it is. Yeah, you're right. It's outcome harvesting and it's only done annually. So that means you get an attitude change, you get it in the right direction, but they only measure it once a year.
So there's all sorts of different things that could be going on that could change their attitude that's got nothing to do with the project. Yeah, so, yeah, that's a good point. Yeah. Yeah, so suggestions on how this could be improved is definitely, first of all, addressing the regular assessments, right? Implement more pre-project surveys to establish a baseline, conduct regular follow-up surveys to track changes over time.
They could also use a comparison group where people that are not in the project, that are not doing anything to any of these women or this target group to change their attitudes, they could also measure their attitudes on the same variable, which is increased willingness to particularly these commune officers to address the issue of women's unequal caregiving burden.
And they could go to other districts in Vietnam and ask commune officers there where Care Canada is not working and trying to change their attitudes. Their attitudes could also be going in the right direction. Right, right. Yeah, that's a key point. And one of the methodologies I have here to note that would expand on the only tactical assessments of this particular indicator is, one, they can also implement longitudinal study design methodology that measures attitude over multiple time points.
And this would allow for assessments of changes in attitudes and the sustainability of these changes over time. Another thing that they can do is implement a mixed method approach where they incorporate qualitative methods such as focus group discussions or in-depth interviews. And that is complemented with the quantitative self-reporting data. And that would provide a bit more context to the attitudes expressed and the reason behind any changes or lack of any changes thereof.
Yeah, that's right. There's a common problem. Our focus groups are used just to measure the outcome indicator rather than measuring the outcome indicator first and then seeing that the outcome indicator is not moving in the right direction. And then, and only then, should they have a focus group where they ask the question, hello, why did your increased willingness not go up? And then you hold a focus group to find out.
But often focus groups are abused, misused, and they only use them like outcome harvesting to try to measure the outcome indicator in the first place. Right, correct. Okay, so let's move on to the next outcome, which is increased ability of ethnic minority women to meaningfully participate and make decisions related to economic activities at the household level and in their livelihoods. And Mark, I think you're gonna be dealing with 1120.1 indicator. Well, yes, I'm reading.
Percentage and number of ethnic minority women who have meaningfully participated in economic decision-making at the household. Yes, I think general increased ability participation are difficult terms to evaluate and also to monitor because meaningfully participation, what does it mean in detail? Does it mean, that they have not the chance to participate in the household before? Is there, because of what? Because of cultural aspects or social aspects.
And now if they are able to do this, it's, we can maybe, this is for me also a question of a qualitative approach necessary, a question of qualitative measurement. And general decision-making, you have to go back and ask them, maybe in interview situations, what has improved your participation at the household? Is the husband now allowing you, or is your role in the family has been stronger than before to explain this indicator?
And the other way is, sometimes the stories of change can also help to get to the point where you get to verify and to find out what is now the improvement of the situation. And the second one is similar, it's the percentage of numbers of ethnic minority women who report participating and voicing out their concerns about pay economic activities to local authorities and DTPers here. Here we have a similar situation.
There's one aspect, it was written women who report, they're reporting their participation and voicing out their concern. So it's a self-reporting or it's a reporting that happens via a survey, I don't know. But for me, when I look at this thing, what local authorities are they, what type of DTPers are these who are concerning about this? Are they from the district, from the communities? And how can they see improvement to the situation before?
It's a little bit tricky to use such indicators who are, you can use your qualitative approaches to by the end of the day, it's to see the change from the baseline to the target. It's sometimes not so, we will not know it's not so visible and also hard to measure it, yeah. Right, and if you look at the outcome, it says we've bolded it for, if anybody's interested in getting the performance measurement framework, you just email me, it says increased ability to meaningfully participate.
Now you could argue, you can measure that quantitatively. You could measure their ability to negotiate or whatever, you could test them, you could do some sort of assessment before, during, and after the training instead of self-reporting. So that's a really quick way of arguing you could actually objectively measure increased ability based on some sort of training where they're training them how to negotiate for better participation in the household.
And that would be good enough because the outcome is clear. It just says increased ability. After that, good luck. We don't know if that's gonna lead to greater levels of satisfaction for these women in the household, but at least that piece, that short-term piece in the immediate outcome of increased ability, they could measure it. But they're going straight to stories of change.
And to be fair to Care Canada, of course they're doing what Mark has explained, which is trying to find out how to measure this level of participation in a household, which is very difficult. It's intrusive. So we understand stories of change may be a way of doing it, but you've forgotten your outcome in the first place, which is increased ability.
So they've kind of missed, in my opinion, that first step where they could actually measure their ability using some sort of psychometric test of their skills to be able to negotiate based on training they have received in the project. And then you could also do this with another group of women who are not even in the project or in some other part of the country, and that would be good enough. They could show that ability has actually increased.
The other question is also when you talk about household, then you have also to ask other members of the household, the husband or children or people who live in this household if the economic decision-making has been improved because of the project. So you need more voices to understand what is really the improvement.
Even though they've made it clear in that second indicator, they only are interested in the women, but it's a good point is that it's a more complex dynamic in the decision-making within the household. But they say right in the, at least I give them credit for specifying, it's the ethnic minority women. They want to increase their participation. But you're right, it may depend on the other members in the household, absolutely, yeah.
Okay, so we're gonna move on to the next outcome, which is in the performance measure framework, increased capacity of civil society organizations, including women's rights organizations, private sector companies and media organizations to promote poor rural and ethnic minority women's economic rights in Vietnam.
And the indicator, number of civil society organizations, media organizations and network members, private sector companies, who report an improvement in their perception, opinion related to ethnic minority women's economic rights. So again, outcome is clear, increased capacity. So that means we should be looking at their ability, increased ability to actually advocate for these women regarding their economic rights.
So there's a huge bias in here because these organizations are paid for by Care Canada, by the project to advocate for these women. So they're expected to advocate on their behalf. So if you ask them, have you improved your ability or your perception or opinion related to women's economic rights that you happen to be paid for by the project to advocate? Of course, they're gonna say yes. So this is a huge bias.
And again, they missed the point of the outcome, which is they should be just measuring their increased capacity to advocate. There's all sorts of projects where they train people on how to advocate for another group, which is what's going on here in this outcome. So that's the major problem I see with that. And I pointed it out, why not objectively measure and also the other point is it's not the institutions, it's the human beings inside of them that we need to look at.
We need to measure their technical capacity to advocate for others, right? So it's not just the number, that's the real problem here. The indicator says the number of organizations. So you just tick a box off. How do you tick that box off that says this civil society organization in Vietnam has met all our criteria of equity of adequately and increasingly in better ways advocating for these women? How do we know?
So what we need to do instead is go inside that organization and test a group of the individuals who are expected to advocate for those women. I think it could be duty bearers, I'm not sure. And make sure that their technical ability to advocate has gone up and maybe even their attitudes have gone up.
And then the other problem of course is even if they have gone up, there may be another group of civil society organizations that are doing just as good a job or even better job at advocating for ethnic minority women's economic rights in some other part of the country where Care Canada is not involved at all. So that's the problem with that indicator there. Now we'll move on to the next outcome which is increased participation by poor and ethnic minority women in paid economic activities.
I think Mark you're doing this indicator here. Yes, percentage and number targeted poor, rural and ethnic minority women and men who report access to and control over economic resources desegregated by sex, ethnicity, age, disability, status and commune. Well, here we have again the same question like we had before, it's about self-reporting. Because you want to know this ethnic minority women and men who are reporting their access and control of our economic resources.
And it's somehow difficult to value it. The measurement is somehow you can maybe check them in a questionnaire to see what economic resources they had before the project, what do they have now? So you can see the change in the impact but there's always some bias about self-reporting but it's sometimes not bad.
It can also be if you do it in a way with interviews and focus group discussions to try to get the view of the people and to put it in numbers and figures it's time to see the track changes that it's a little bit not appropriate I think. You can maybe track their incomes. You can track their incomes against a non-project group of women and men to see if they have really increased their incomes before. Exactly, yeah.
Now to be fair to Care Canada, the outcome is increased participation but that's difficult to measure but they've said they're gonna claim that their project is going to increase the levels of participation for these women and they're gonna train them in some way that this is going to happen through a variety of project services that are described in part one.
But yes, it's difficult but yeah, you could just cut to the chase and figure out because you want to achieve ultimately increased incomes, economic resources for the family although we can't assume that all that money, increased income goes into the women's pocket may still be held by the husband and that is understandable. That's why they talk about empowerment of the women specifically. So it's difficult but I think we've noted it's a challenge for sure in measuring increased participation.
Absolutely, yeah. Yeah, it's a general hard aspect. Participation is generally very difficult in developing projects so very often you use quantitative approaches because quantitative, you can work with a questionnaire and make a survey if you have a reliable sample but it's not always giving you the right view. Right, okay good.
So the next outcome is enhanced promotion of economic rights for poor rural women, especially ethnic minority women in Vietnam and the indicator I'm looking at here is average total number and proportion of daily hours spent on unpaid domestic and care work.
Now this is an interesting one because they are trying to have these women, part of the project actually delivers childcare saving options and they don't describe the options but we can assume that just by delivering that service in the project which is very important, it will reduce the hours that these women spend on unpaid domestic care work just by providing these alternate options for them to choose, which is fine.
Two problems again, the one is even if that happens, we need to see a comparison group where there's another group of women who don't receive these alternate options of other people taking care of their children for example in this project.
In another part of the country where another group of women get together for example and set up their own community childcare center where one woman says all your children can come to my house this day and the rest of you can do what you have to do during the day and I'll take and they do that on a rotational basis. So that's the first problem. The other problem is when you look at the outcome, again, they're looking at enhanced promotion of economic rights for poor rural women.
So this particular indicator isn't promoting their rights specifically so enhanced promotion is not really being measured here directly. So that's the only way I could look at this is saying that is there another group out there that are also promoting the economic rights of these women in a way that's having the same benefits. So there's no comparison group. So that's the problem there with that indicator.
Now to be fair to them, they have other indicators where they are trying to measure enhanced promotion but we're not going to discuss that right now because we have limited time to cover all the indicators but the summary is there for people to look at as to why all of the outcome indicators do fall into one of those five problem areas that we're discussing.
So the next one, the next outcome is improve economic well-being for poor rural women, especially ethnic minority women in Vietnam and the indicator there is percent and number of poor rural and ethnic minority women and men who report an improvement in their economic well-being. And it's the same issue we've talked about before, which is instead of having them self-report on their economic well-being, why not just measure their incomes?
And more importantly, do it compared to another group of women in another part of Vietnam that aren't receiving all these project services and track their incomes before, during and after the project and see if there is a significant increase or greater increase in incomes for the project group. So now that we've done that, we've covered all nine outcomes of the project and picked at least nine outcome indicators. Let's, I think Mark, thank you.
You've already covered the other indicator for that outcome, but we'll just do one more. And that is, we'll go back to increased entrepreneurship capacity of women-led grassroots social enterprises to pursue economic opportunities in Vietnam. And we'll look at another indicator. Donald's gonna look at 1230.2. Yes, thanks David. 1230.2, this indicator states, percent a number of ethnic minority women-led grassroots enterprise members who report a change.
And here they're using a five-point scale in their confidence level of their entrepreneurship capacity. This has the same issues as the 1230.1 that we discussed, or issues that relate to increased capacity, how that's being defined. Issues with self-reporting, the reliance of it being self-reporting. And the major issue I think that I have with this indicator is confidence assessment.
To measure confidence and to improve on this indicator, they should probably use a psychrometrically sound instrument that would differentiate between perceived self-efficacy and overconfidence. And these often do not correlate with actual competence. So there are some clear technical issues with this particular indicator. And to improve it, those issues will be need to improve. Define what is capacity. Is it financial, market assessment, operational, and so forth.
Address the issue with confidence, competence, and that would clear up the major issues with this particular indicator. Before we make a decision on this performance measurement framework and its ability to support Care Canada in its claim that its project outcomes have been achieved by its project services, I'm gonna go again to the last indicator on the last outcome, which is improve economic well-being for poor rural women, especially ethnic minority women in Vietnam.
They have one other indicator, which was level of improvement of economic rights as perceived by ethnic minority women. And that's fine. It's a good way to look at, measure their perception levels to see if their perception levels of how they're doing economically, it's obviously subjective, but there's nothing wrong with that. It's not a bad indicator. But again, the problem is there's no comparison group.
So we need to get another group of women outside of the project, some other part of the country in Vietnam, and measure also at the same time their perceived levels of economic well-being where the project's not operating. And if this project wants to claim that they're more successful, then they have to show that their levels of perceived economic well-being have increased at a greater rate or are significantly greater than the non-project group.
So now that that's done, what do you think guys, after we've reviewed this performance measurement framework, do you think that Care Canada can make the claim that its project services are achieving its project outcomes at an expense of $3.4 million to the Canadian taxpayer? Donald?
Based on the analysis that we just provided and the limitations that we identified with the indicators, I think it would be extremely difficult for us to conclusively claim, or Care Canada to conclusively claim that the program has achieved its objectives. And those are based on key issues we discussed, self-reporting bias, lack of objective measures, no baseline or control groups, measurement frequency issues, vague definitions, and no link to specific activities at all.
With all these issues, it's very difficult to conclusively claim that the program has achieved its objectives. Good, Mark? Well, I think in general, I have no... The issue is that for me, it is not stopping at economic rights. You also consider social and cultural rights, because in the end, you want to include them, the rights to have adequate food, adequate housing, adequate education for minority groups.
And to measure that in this template, to measure that in this type of project, it seems for me complicated, and they should be interlinked also, like I said before, the economic, social, and cultural aspects should be interlinked. And one point is that if you want to see improvement, you have also to talk with the authorities without duty bearers, if they have measures, if they have laws or policies that can improve the level of ethnic minorities in this district or in these places, yeah.
Yeah, that's great. I think I mentioned in part one, that issue where the duty bearers are supposed to advocate for the women, but the bias is they're asked how they believe themselves are doing in that role, and it's not objectively looked at by a third party. And you need to look at that, I think. So that's good, okay. So we basically summarize that CARE Canada needs to do some work on its outcome indicators for starters, along with a lot of other issues.
So what we're gonna do now is send this part two to the Minister for International Development in Canada, as well as the shadow critics for the Conservative Party and the New Democratic Party, and we'll also copy CARE Canada. And for you listeners out there, send me an email at evaluatecanadaaid at gmail.com if you want a copy of the performance measurement framework and a copy of the Excel summary of all the outcome indicators and why they fall into one of these five problem areas.
We're also looking for evaluation experts who may wanna be interested in doing similar work to what Donald and Mark have done today. Send me an email, and I'd be happy to send you other performance measurement frameworks that I'm looking at for future podcast episodes. And if you have any comments or suggestions on this particular episode regarding CARE Canada and our review of their performance measurement framework, send me an email and I'd be happy to respond.
And at some point, we'll be putting the performance measurement frameworks and the summaries of their outcome indicators on a website.
And finally, for you listeners, Canadian taxpayers, you may wanna email CARE Canada at info at care.ca and simply ask them are they gonna be improving their outcome indicators, or you could also ask them if they're gonna put all their other project performance measurement frameworks on their website so that we can all have a look, because right now they're not available to the public.
So thank you for listening and stay tuned for an upcoming new international development organization, $37 million project, so it's getting a little heavier. Thank you, Donald, and thank you, Mark. Thanks, Evan. You're welcome, David. All right, thanks and stay tuned. Bye for now.