Welcome to the Improving Development Evaluation Podcast. I'm your host David Wand and welcome to season 1, episode 1, part 1. In this episode we're going to introduce you to an international development organization. That international development organization is Care Canada. You can learn more about Care Canada if you go to their website at www.care.ca. Their headquarters is located in Ottawa, Canada.
The project we're going to be talking about that Care Canada has been funded by the government of Canada to deliver is called Advancing Women's Economic Empowerment. This project is being delivered in Vietnam and the cost of this project to the Canadian taxpayer is $3,400,000.
And in this part 1 I'm going to give you a brief description of the project services that are being delivered and a critique of Care Canada's performance measurement framework that they have developed to measure the project's performance. That is, the achievement of the project's outcomes using the outcome indicators in that performance measurement framework.
But before I describe the project services that are being delivered for this project by Care Canada and their partners in Vietnam, I'm going to go to my summary and conclude that in this performance measurement framework for this project there are 19 outcome indicators that are measuring 9 outcomes. My review of all 19 outcome indicators has led me to conclude that all of these indicators are not measuring the outcomes.
So the organization, Care Canada, cannot make the claim that it's achieving its project outcomes with the services they're delivering in the project simply because their outcome indicators are not properly measuring the outcomes for this project. As you probably know from the trailer for this podcast, those 19 outcome indicators fall into one of five problem areas.
In this project, 11 of them, those outcome indicators, fall into the how do you feel self-reporting problem where people are asked how they feel about achieving the outcome rather than being objectively measured on whether they achieved that outcome.
Another seven of the outcome indicators fall into problem area number five, which is there's no comparison group of people outside of the project that are also measured on the same outcome, which means even if the people in the project are achieving the outcomes, we don't know if they're achieving them any better than another group of people that could also achieve the outcome outside of the project.
And finally, we have one indicator that is falling into problem number four, where it's simply not being measured frequently enough. If you wish to receive Care Canada's performance measurement framework for this project and my summary of the outcome indicators for this project, send me an email to evaluatecanadaaid at gmail.com and I would be happy to send those documents to you. I plan to post all of these documents for each episode on a website that is currently being developed.
So now that I've concluded that the PMF is flawed, let's get into more detail about the project services. So there are four target groups that this project delivers services to. The first target group are poor, rural, ethnic minority women, and the services being delivered to them are training on negotiation and household decision making, action plans being developed for them to promote their economic rights and are promoted by male champions.
Dialogues are held between these women and men on household decision making and gender based violence and unequal caregiving burden. These dialogues are also held on the same topics between these women and communal officers. Community events are also held, attended by women and men to promote the economic rights of these women. And gender based violence services that are available for these women are promoted to these women.
And finally, a feasibility study is completed on child care services and household time saving appliances. And then there's a variety of trainings that are also delivered to these women, such as training on financial literacy, savings group operations, digital record keeping, and business plan development. These women also receive unpaid care time saving options. And also studies and policy briefs are produced on the benefits of reducing unpaid work for these women.
Training is also delivered to these women on climate appropriate agricultural practices and on high quality tea production and safe migration. A second target group in this project delivers services. And that target group are civil society organizations and media organizations. The services they provide are they promote the economic rights for these poor rural ethnic minority women, such as including success stories of women entrepreneurs, which are promoted in the media.
These civil society organizations and media organizations also attend learning events paid for by this project, where these civil society organizations and media organizations are expected to learn on how to work with these women whose economic rights they're promoting. These civil society organizations and media organizations are also expected to produce documentation. During the promotion of these women's economic rights, this documentation is sent to policymakers.
Also round tables are organized by this project between these civil society organizations and policymakers on how to promote the economic rights of these women. The third target group are private sector individuals. And they receive from this project training on women's economic rights and gender responsive value chain approaches. And finally, a fourth target group are what we call duty bearers or what they refer to as village leaders and communal officers.
The services the project delivers to them is training on gender responsive programs for women. So that gives you an idea of the services the project delivers. Now, there's nine outcomes in this project. And they are outcome number one, increased entrepreneurship capacity of women led grassroots social enterprises to pursue economic opportunities in Vietnam.
Outcome number two, increased gender responsive capacity of poor rural and economic minorities, especially women to save and borrow money to support productive and sustainable economic activities in Vietnam. Outcome number three, increased gender responsive capacity of poor rural and ethnic minorities, especially women to earn income from climate smart and sustainable on farm and off farm livelihoods in Vietnam.
Outcome number four, increased willingness of poor rural and ethnic minority women and men duty bearers, in particular, commune officers to address the issue of women's unequal caregiving burden and economic rights in Vietnam. Outcome number five, increased ability of ethnic minority women to meaningfully participate and make decisions related to economic activities at the household level and in their livelihoods.
Outcome number six, increased capacity of civil society organizations, including women's rights organizations, private sector companies and media organizations to promote poor rural and ethnic minority women economic rights in Vietnam. Outcome number seven, increased participation by poor and ethnic minority women in paid economic activities. Outcome number eight, enhanced promotion, economic rights for poor rural women, especially ethnic minority women in Vietnam.
And finally, outcome number nine, improved economic well-being for poor rural women, especially ethnic minority women in Vietnam. So we can't go into all the outcome indicators, but I'm going to give you just three of those outcome indicators as to why they fall into one of those five problem areas that we outlined in the trailer.
So in part two of this episode, where Care Canada will be invited to respond to this critique, we will cover all nine outcomes with at least one outcome indicator to be discussed for each outcome. If we have more time, great, we'll go into further outcome indicators, but we're going to focus on making sure we cover at least one outcome indicator for each of the nine outcomes so you get an idea of right across the entire spectrum of outcomes, the nine of them for this project.
At the end of part two, the evaluation panel, that's me and two evaluation experts, will be concluding as to whether Care Canada can make the claim that its project services are achieving its project outcomes. As host, I conclude Care Canada cannot make this claim, but I want our listeners to hear what Care Canada has to say as well as the other evaluation experts.
So let's look at the outcome increased capacity of civil society organizations, including women's rights organizations, private sector companies, and media organizations to promote poor, rural, and ethnic minority women's economic rights in Vietnam. So that's the outcome.
One of the indicators to measure this increased capacity to promote is number of community, civil society organizations, media organizations, and network members, private sector companies, who report an improvement in their perception, opinion related to ethnic minority women's economic rights. You probably can tell already from this indicator that it's flawed, but it's interestingly flawed in two ways.
First, reporting on the number of civil society organizations who report an improvement in their perception or opinion related to ethnic minority women's economic rights, it's got nothing to do with increased capacity to promote their rights. They're just asking them for their opinion, but the outcome is increased capacity to promote.
And if you look at the project services, they are expected to improve their capacity to promote because if you remember from the description of the project, they are showing up at a cost to you, the Canadian taxpayer to learning events where they're expected to learn how to work with the women whose economic rights they're promoting. They're also being paid by you, the Canadian taxpayer to promote the economic rights of these women.
And of course, they are going to say they have improved their opinion on promoting these women's economic rights. They're also expected to produce documentation on how they're promoting these women's economics rights with that documentation being sent to policymakers also at a cost to the Canadian taxpayer and are showing up to round tables where they sit with policymakers on how they're going to promote women's economic rights.
So it's clear the project funds are being spent to increase their capacity, technical capacity on how best to promote. But the outcome indicator is not measuring that at all. It's just measuring how they feel, their perception, their opinion. We don't want their opinion. We don't want their perception.
We want to know if they're technically capable, thanks to all that money spent showing up at these learning events, showing up at those round tables, producing documentation that shows they know how to promote the economic rights of these women. And more importantly, that they've significantly increased their capacity. So the indicators got nothing to do with the outcome. That's the first problem.
Even if capacity is defined not as technical capacity to promote, but rather an improved willingness to promote, they are going to report yes, because they are paid by the project to promote. And even you can go a step further. Let's say we've got an objective measure that they've improved their willingness.
There could be another group of civil society organizations out there in another part of the country that have got nothing to do with the project and their willingness to promote has also gotten. So we'll never know on several levels whether these organizations have increased their technical capacity or even their willingness to promote.
Another indicator we're going to look at is for the outcome enhanced promotion of economic rights for poor rural women, especially ethnic minority women in Vietnam. One of the indicators for that is the average total number and proportion of daily hours spent on unpaid domestic care work. Now if you remember from the project services, one of the things they deliver to these women is unpaid care time saving options.
We don't know what those options are, but we suspect and we logically expect them to reduce the number of hours spent on unpaid domestic work for these women. If they're given these unpaid care time saving options. Unpaid care time saving options. It means they don't have to spend as much time on unpaid care themselves. Sounds good to me. Problem is we don't have any comparison group.
What about another group of women who are not in the project in some other part of the country in Vietnam who could say, ah, let's get together and set up a communal child care center. We'll all send our children there and just one of us will watch all the children together collectively while the rest of us reduce our hours of unpaid child care. So we also as a group perform well on the outcome indicator with our hours also going down like the project group of women.
So there's nothing wrong with the indicator, but if Care Canada wants to claim that their project service of delivering these time saving options is making a greater reduction in hours spent on unpaid child care by this service, they have to compare it with another group outside of the project. And finally, one more indicator we're going to look at is related to the outcome improve economic well-being for poor rural women, especially ethnic minority women in Vietnam.
One of the indicators there in the performance measurement framework is percent and number of poor rural and ethnic minority women and men who report an improvement in their economic well-being. Now, we don't know exactly what Care Canada asks them in terms of economic well-being. So to be fair, maybe at the podcast part two, they'll bring up something like a world bank survey or something, but we just don't know.
So what I'm suggesting here is maybe they should measure their incomes or measure their assets rather than asking them to self-report whether they feel their economic well-being has improved. Maybe it's too intrusive, but I would think measuring their incomes or assets over time before, during, and after the project would be a better measure of showing that their actual economic well-being has improved.
That world bank survey that you often hear about with the percentages of those living on $2 a day or less comes to mind. And again, even if their incomes or assets increase for the project women, what about a comparison group of women not in the project and measuring their incomes or assets as well over the same time period so you can compare the two groups. So those are just three examples from three outcome indicators that fall into these problem areas.
If you have any comments or suggestions for this episode, feel free to email me at evaluatecanadaaid.gmail.com. And if there are any evaluation experts out there who'd like to be on the podcast to look at other performance measurement frameworks and in particular the outcome indicators, just like I have here, email me and I'd be happy to send you the remaining performance measurement frameworks that we have lined up for season one where we don't have evaluation experts so far.
Right now that's eight performance measurement frameworks that we're looking for experts to comment on. What we're going to do now is invite a Care Canada representative to attend part two of this episode to respond to this critique here in part one, along with two evaluation experts that will attend part two of this podcast. Thank you for listening and stay tuned for Care Canada, part two, episode one, season one.