Welcome to the Improving Development Evaluation Podcast. I'm your host David Wand and welcome to Episode 3, Part 2, where we continue our discussion about the United Nations Population Fund South Sudan Project, which is called Human Resources for Sexual Reproductive Health in South Sudan.
As you may recall from Part 1, I reviewed 16 outcome indicators covering 7 outcomes and came to the conclusion that the South Sudan UNFPA Country Office could not make the claim that they were achieving their project outcomes simply because their outcome indicators were not properly measuring their outcomes. On November 18th, I sent to the UNFPA South Sudan Country Office through their contact form an invitation to attend the podcast today on December 14th.
But as of today, we have not received a reply from UNFPA. I also sent it the same invitation to the headquarters in New York, HQ at UNFPA.org, as well as the East South African Regional Office at isarro.info at UNFPA.org. And I also sent it through a contact form at UNFPA.org in New York, indicating it should go to the evaluation office in New York. Even despite those four contacts, I haven't received a reply from any of them regarding this invitation.
But fortunately, we do have an evaluation expert with us today from Accra, Ghana. His name is Benjamin Serbo and he holds a master's degree in political science from the University of Zululand in South Africa. And he is a monitoring and evaluation specialist. And we're glad to have him on the podcast to provide more insight into the flaws in the performance measurement framework for this project that the UNFPA is delivering in South Sudan. Welcome, Benjamin. How are you? Yes, I'm good.
Thank you. And yourself? I'm fine. I'm very happy that you could join us. So what we're going to do now is we're going to go through each of the seven outcomes and to make sure we cover the whole spectrum of the project. And then if we have additional time, we'll go back to some of the other outcome indicators.
So starting with the outcome in this project, improved national and state government capacities to develop and implement sexual reproductive health, gender-based violence, HIV policies, legislation and coordination mechanisms. And Benjamin, you're going to be taking a look at an outcome indicator for that outcome. Okay, you're right. Thank you very much. So the outcome indicator for the number of state coordinating mechanisms and coordination plan. David, is that correct? That's correct.
So first of all, let me say this, when you talk about the state coordinating mechanisms, it is not only the project which is going to develop the state coordinating mechanisms. And the number of guests who are going to develop this mechanism. And so the project cannot claim that. If you're not paying those state mechanisms for it, then you can sue the project. So then you need to look at the kind of contribution that the project made.
So for instance, when you look at having an hour with black, that's not necessarily mean you supported or you did it, you supported it. As with either some of the areas that the project will have to look at. Now, one thing we need to know is that when you talk about having a state coordinating mechanism, and I know it will not be able to justify how the project was continued, improvement of state capacities.
Rather, I would like to see the number of project officials who contributed in the development and the implementation of those annual coordinating. Because when you are conducting an evaluation of the project and you know how the project contributed to improving state and national capacities, you have to assess how the beneficiaries of your project played a direct role in the development of those mechanisms.
So one of the indicators we can look at in this, for instance, the number of project beneficiaries who participated in the implementation of the annual coordinating. So assuming the five of your project beneficiaries participated, then we can get some qualitative information that can support the provided to the development. So in that case, we'll be able to know the contribution that our project made to strengthen the state. So these are some things that we can look out for.
But if we just did the number of state mechanisms, annual coordination, yes, we can have the number. The project will not be able to clearly demonstrate how it contributed to the development of the mechanisms or development of the annual. That's a good point. And also it seems to be misplaced, the indicator. It's not quite right, let you say, is focusing on who there's no who in here.
And so, yeah, it should absolutely be specific to the project participants who were trained on developing these mechanisms. Yes, that's a good point. And maybe you're going to say the next part. So they also, maybe I will just add, if you look at the outcome, it's very clear that they want to improve capacity. And just saying number is not good enough. You have to show that these mechanisms and coordination plans have actually been improved in terms of quality.
So once you get the project participants involved, then you have to show that from the trainings that the project participants received, they actually improve the quality of these coordination mechanisms or annual coordination plans. Perhaps you were going to say that next. Yeah. Okay. So that's great. So it really actually shows two things that are missing. It's misplaced, the indicator, and it's not measuring increased levels of capacity. That's great.
So let's move to the next outcome, which is strengthened capacities of health training institutions to produce quality human resources for sexual reproductive health rights, gender based violence, and HIV services. And Benjamin, I think you were going to look at an indicator there. Okay. So with this indicator, with this outcome, they indicate in the outcome proportion of targeted health training institutions, MOUs signed with targeted practical site.
And the number of supportive machines conducted at a health teaching. With this outcome also, now let's indicate the proportion of targeted health institutions, MOUs. Strengthening MOU does not necessarily translate into strengthening the health institution. It doesn't show any, because MOUs now by health institutions has strengthened.
So then when we talk about strengthened capacities, we are looking at how the training institute participating in the project have improved their structure, process, human resources to deliver quality training to various healthcare providers. So just finding MOUs conducting supportive visits, that's not the most ridiculous. So we're looking at these two indicators. We cannot use it to measure the strength and capacity of the health institute.
So and liberal you rightly mentioned, we can measure the achievement by looking at the percentage of faculty that's reached certain level of technical capacity. So in that case, the UNFPA considered a freeway index. Number one, can you get a percentage of teachers or trainers of beneficial health, who meet the required national or international certification for expertise. Usually we have this certification which can be used by the national.
So in order for this to be strengthened, we need the people working in this institution to find all the expertise. So then in that case, we can say our institution, the institution has strengthened with the staff achieving these issues. Even that's one indicator. Now another indicator you can number health training institutions, who are currently meet the required industry standards. So I believe the training curriculum for the HRHRN keeping as opposed to meet certain standards.
Obviously there will be criteria to determine it meets the standards. So the institute meeting the standards be a very good indication of its strength and capacity to deliver quality training and produce quality human resources. Actually this is another indicator, those can be considered. And then other areas that can be considered are the satisfaction of the health care that will receive training.
And even the beneficiary of the gender based fund is and then sexual and reproductive health service from this health care by this because when you get a fee from the institute, they will satisfy the kind of service that we provide. It's going to help you to mention or to determine whether the institutions are strengthened and whether these institutions are providing quality, GVVN, HRHRN services to the brand and to the people. So these are the indicators. Thank you, Benjamin. That's great.
And so what we know here is that the indicator in its current form is not measuring the outcome at all. At all. It is not measuring the outcome. That's right. Yeah, an MOU signed is clearly not a measure of strength and capacity. Great. Okay. So we'll move on to the next outcome, which is increased awareness among women and girls to utilize gender responsive, sexual reproductive health, gender based violence and HIV services. And there are two outcome indicators for this outcome.
And I chose the second one, proportion of women, girls from targeted communities who report making their own decision for seeking sexual health services disaggregated by age. I correct myself, Benjamin. I've seen my notes. This is yours. Sorry. Go ahead. That is mine, right? The outcome indicator is proportion of women, girls from targeted communities who report making their own decision for seeking sexual health services disaggregated by age. So we are talking about increased awareness here.
So when you talk about it, increased awareness, it will mean the procedures and processes they need to give you the HRH services. So then this could lead to reported cases. And this is something I will see. It was awareness to resolve a number of cases reported. They understand the processes, you ensuring that their awareness has been treated. There are GBV cases. So now let's look at this. With the original indicator, you don't have any comparison in there.
So now your project is getting its awareness. It can happen to the natural population somewhere else. And then they also go in a student. So then if you have a comparison, you will be able to extend to which your project made the difference. So with it, current state without Paris, it will be difficult for the project to make the contribution to creating awareness. So in order to ensure that we are to be confident, the project is strong, patient awareness.
We need to make sure that the patient is aware of the patient's awareness. Yes, thank you, Benjamin. And if you look at the performance measurement framework for this indicator, they do measure it three times, baseline, midline, and I think they call it end of the project. And you'd think, oh, well, if we can show that the percent of women reporting this has gone up, that's good enough, but not necessarily like you pointed out.
So here we have in the performance measurement framework, three measures for this indicator, a baseline, midline, and end of project. But even if the percentage of women who report utilizing services has gone up, which is good, it could be going up even greater somewhere else in the country, like you said, where the awareness campaign and money spent by this project to increase awareness like radio ads, brochures.
I'd have to go back to part one to find out exactly what I said, how they increased awareness. But still, you need a comparison group to show that the project is doing it better. So yeah, that's a good point. Okay, so we can move on to the next outcome, which is strengthening capacity of targeted national and state level health facilities to provide quality, integrated sexual reproductive health, gender-based violence, and HIV services.
And the indicator I looked at here, there's only really one, it's a cluster of, I think, three. And those three are number of visits for family planning services in targeted locations, number of visits for antenatal clinics in targeted locations, and number of deliveries by skilled birth attendants in targeted locations.
And the problem here with these indicators for this outcome is they're not really measuring, as we've discussed earlier on another outcome, they're not measuring strength and capacity of the health facilities.
And if you look at part one or listen to part one and look at the project, the number needs to be changed to the number and percent of targeted facilities that have solar power, have water supply, have adequate staff trained to maintain the solar and water supply that have sufficient essential health workers that can deliver gender responsive sexual reproductive health services, gender-based violence services, and HIV services. So the number is just inadequate.
You need to go a step further and figure out the percent of staff that actually have the capacity to provide the services in a minimum quality way, because it says quality integrated services. So just counting the number of visits or the number of deliveries or the number of visits for family planning or antenatal services is not good enough. And that's why it falls into the problem area of not related to the outcome of showing strengthened capacity of these targeted health facilities.
And the other point is, even if they modify the indicator to have a criteria of what is an acceptable minimum percent of staff trained at each facility that knows how to repair and maintain the solar power panels and the water supply, as well as having a minimum number of staff that have adequate skill levels to deliver the services, you could argue that in another part of the country where the project's not operating, they've got even a higher percentage
of health facilities that have strengthened their capacity to deliver services. So that's basically the issue for that outcome indicator. Now the next one is the next outcome is strengthened enabling environment at the national, state, and local level for women and girls access to sexual reproductive health and gender based violence services. And we've talked about this earlier, so it kind of overlaps. But it's the same issue.
The indicator here is the proportion of targeted health science institutions that meet the minimum educational standard for training health professionals. Now that's an excellent indicator. But the problem is we don't know inside of these institutions what's going on. And in the performance measurement framework, they actually talk about the staff that train these health professionals being assessed. And there actually is progress reports based on those assessments being produced.
But we don't know what sort of measurement tool they're using to measure these staff, to say that they've reached a minimum educational standard, and then for that one health institution, we don't know if that's sufficient to conclude that the health science institution all by itself has reached the minimum educational standard. And like you had mentioned earlier, Benjamin, another indicator could be even looking at the curriculum content to make sure it's up to a minimum educational standard.
So unfortunately, UNFPA are not here to explain. And I've looked at their evaluation plan, and there's no measurement tool there talking about how they come up with this percent. This is despite the fact that they have a target of 80% of health science institutions reaching this minimum educational standard. So we need to ask the UNFPA, how are they measuring the staff?
And coming up with this calculation to conclude that this is the percent of targeted health facilities that meet the targeted health science institutions that meet this minimum educational standard for training health professionals. We simply don't know how they calculate that.
So it's not adequate or related to the outcome of saying we have strengthened the enabling environment for women and girls access to services, because we assume when they graduate these midwives, that they're up to standard, and that the women and girls who get services from them have experienced a strengthened enabling environment. But we just don't know.
The next outcome is increased use of gender responsive sexual reproductive health rights and gender based violence services by women and girls, especially those living in vulnerable situations. And the outcome indicator for that is the proportion of women aged 15 to 49 who receive four or more antenatal visits. Now this again has issues here where they clearly have targeted sites per targeted locations.
So we need to measure these same indicators for areas in the country of South Sudan where the project is not operating so that we can show that the proportion of women who receive four or more antenatal visits is higher in the project area for this project, $37 million, compared to health facilities that haven't been targeted. Because one of the services is actually to deploy midwives from these institutions to these targeted health facilities.
So we would expect the percentage of women in that catchment area for that targeted health facility to get greater access. So the only way to know that is to have a comparison group, and we don't have any comparison group mentioned in this performance measurement framework.
So that's something that's important to recognize is that it's really important to have comparison groups when they are even explicitly saying we are targeting particular locations, which means other locations we're not targeting. So the project should be showing that in the targeted locations, they're getting greater rates of service where higher percentage of women are receiving four or more antenatal visits. And the final outcome- And David, if I can- Yes, jump in.
I mean, yeah, usually when we get to the outcome level, I usually want to see some kind of feedback from whoever is benefiting. So for instance, we are looking at proportional age 15 to 14 who receive or antenatal visit. Assuming they receive this for more than one, so then on the outcome level, I would like to see those women who are receiving this, they can observe it. They receive from this, but to what extent are they satisfied with this kind of service?
So in a matter of fact, they receive the services that normally are not required to function. Then we want to high number of women or patients receiving this to express satisfaction. So if we have like only 40% of them being satisfied, then we have a problem. So then on the outcome level, these are areas that we can look at. Those who are receiving the service, they are at a level of satisfaction with the service. Yeah, David, back to you.
That's an excellent point because it reminds me of a point I missed. The indicator actually says increased use of gender responsive sexual reproductive health and gender-based violence services. So that implies that they not only are receiving the service, like you said, but they're satisfied that the services being delivered are gender responsive.
And that's a big focus of the current government of Canada in a lot of these projects they fund is not only getting them trained up to capacity to deliver the service, but that it's gender responsive. And that's very precise in the outcome. And you're quite right. In the outcome indicator, they have missed that. So that's a very good point.
And that's also needs to be noted in the indicator, not only just the proportion of women who report getting the service, but the proportion of women who report satisfaction with the service being delivered being gender responsive. Yeah, good point. And the last outcome of the project is as we go up the results chain is the increased realization of sexual and reproductive health and rights of women and girls in South Sudan. And the indicators there are excellent. They're universally agreed upon.
The two indicators are maternal mortality ratio and the unmet need for family planning. And UNFPA is very, very good at measuring these. They do large surveys using standard public health surveys. But the problem here is the one we mentioned earlier is how do we show that the $37 million that the Canadian taxpayer paid for this project has reduced the maternal deaths at a lower rate than women and girls who unfortunately did not benefit from this project.
Now UNFPA may come back and say, it's a universal project. It goes all across the country. But my understanding is no, because they mentioned targeted locations. So that means there's an opportunity to pick a non-project group of women and girls and measure their maternal mortality ratios also. So otherwise you don't have evaluation.
You just have a charity project delivering necessary services, but we don't know if they made a statistically significant difference in reducing the maternal death ratio compared to a group of women and girls that did not receive the project services. So yeah. Any comment you want to make Benjamin before we? No, and I really agree with you on this point. Because when you are looking at maternal mortality ratio, the project is supposed to reduce maternal mortality ratio.
So while we are having the project, we expect that by the end of the project we will start seeing a reduction in maternal mortality ratio. What about the other areas where the project is not being implemented? Is it the same? Maybe they will even do a better project. So then you ask people how they want to support that, right? Even though they are not receiving any support or they are not receiving any intervention, but we are still having a reduced maternal mortality ratio.
And that's why it's always important to have the reason. Because it can help you to think about the strategies that you are using to implement your project. If there is something that you should have done, which is done some way, so I agree with you. That's why it's always important to have a comparison group. It helps you to learn the project. Thank you, Benjamin.
I mean, to be fair to UNFPA, another option is they may say we don't have the funds to do a large scale comparison group measuring these people outside of the project. The other thing they could do is measure more frequently just the project group, before, before, before, during, during, during, during, after, after, after. So this is a standard thing that people try to do, as I mentioned in my trailer of this podcast, where you can sort of get around the problem of not using comparison groups.
It's quasi-experimental, they call it. So there's a whole bunch of ways you can maybe measure more frequently rather than just, you know, like they mentioned, baseline, midline end of project. And you could maybe even do small samples of those women and girls more frequently throughout the project. But there's really never any discussion in the evaluation plan about this problem and how to get around it.
It's always the same cookie cutter approach, which is pre-test, post-test only when they're training people, which we know is the same problem, it's not good enough. Or when they get to larger scale measurements like maternal mortality, they just do baseline, midline end of project. And they just assume that they can assume that's $37 million is why the maternal deaths have gone down. So they have options. And the other option I've discussed on other podcasts is just be a charity.
Just say, look, these women and girls need these services, full stop. We're not going to evaluate them. This is an emergency humanitarian situation. And we're going to deliver these services. And we don't care about evaluation because we know we're saving lives by giving them access to these services. And that may explain why there's limited evaluation.
A lot of people are just assuming they're a charity, but unfortunately they're making these claims that they are achieving all these outcomes, but then they're not prepared to evaluate. So in the wider scheme of things, we get it. And we understand that's why I have the podcast to sort of say, maybe you can make a choice here and just be a charity and not make these broad claims of achieving outcomes. That's your other option. So yeah.
Okay. Before we go, are there any other, I think I was going to say we could add, we could talk about other indicators, but I think we've done pretty well here. Did you want to talk about some other indicators or should we just go to the chase, which is sort of to conclude from you at least, what you think of the PMF in its current form? Or do you want to talk about some other indicators? What would you like to do? Because there is another indicator.
There's the number of policy recommendations to government and key stakeholders on health and gender-based violence from that first indicator, first outcome, which is improved national and state government capacities to develop and implement sexual reproductive health, gender-based violence, HIV policies, legislation, coordination mechanisms. There is another indicator there. Number of policy recommendations. Yes. I think I have them.
So policy recommendation, you know, so then is it a recommendation project from evaluation report or my project beneficiary or under development darkness? I mean, the way it has been stated now is to open and for me when I pick, you know, all three, I will not know where do I get the information to measure the number of policy recommendations. Who is making this recommendation? So then it needs to be, yeah, because there are a number of policy recommendations that are made to that.
So then the project will need to be specific. So you can see a number of recommendations made, the project, a result of our evaluation findings or monitoring findings. So of course, we have them planned to monitor and monitor the project. So probably as a result of findings from our evaluation or monitoring, we made this recommendation to the government.
But then when I pick, I will not know, so this indicator is beneficiary, as a result the knowledge they have by making this recommendation to the government. All the evaluation we want, the monitoring of the assessment, as a result of making this recommendation to them. So then this is something that needs to be considered. Actually it's a good point, because if you look at the performance measurement framework, the outcome indicator is buried in the outputs. It's misplaced.
Actually have a number of functional, national, and state coordination mechanisms. So they actually have a good indicator of increased capacity, but it's buried in the outputs. So it's good, but the problem is they just have project reports which report on the quality of these recommendations based on the training that the project delivers to these Ministry of Health staff.
The problem is there's no attribution that the project training has contributed to this increased quality of the policy recommendations. It just says technical support, but it's only done annually, and that's not good enough to show that the training, or the technical support they call it, is responsible for the improved policy recommendations.
So it's buried in the PMF, but it's lower down, and even it's there, even though it's a good indicator, as I mentioned in the summary report that the listeners can get a copy of, they don't adequately measure it to claim that the project training was responsible. It's just an annual report, and they could figure this stuff out all on their own at the same time or in between the trainings.
So that's one of the issues that's common in a lot of these performance measurement frameworks is they do testing to show increased skill knowledge acquisition, but they don't do it frequently enough, or they don't use comparison groups. So there's no way of knowing if the project was responsible, like you were talking about contribution. And a lot of the times they'll say contribution instead of attribution, because attribution is a dirty word, but they're not even there in this case.
So that's a good point. I think we're good. So I wanted to ask you, completely biased of course, the performance measurement framework and its current form, what do you think? Is it adequate for $37 million UNFPA South Sudan project to make the claim that they're achieving their outcomes? In its current state, we say no. So the performance management framework needs to be revised.
Because when you look at the indicators in this case, the original indicators, they had the rain and the rain was that day. So even using the performance measurement framework says your contribution was held in South Sudan. I think it needs to be revised, I mean it needs to be revised. All the indicators, or most of the indicators, needs to be revised again. And those were good for probably the M&A office, the representative from the M&A office of the UNFPA to be here.
I'm not going to say it myself, but you will understand it very well. So it's a minute to us. So at this time, for even the PIAF, we denounce to raise other contributions. Yeah, and even though they're required by Global Affairs Canada to produce an evaluation plan, all of the evaluation plans I've received, including the one for this project, they're all generic in nature.
They don't go through each outcome indicator like we have just done and explain how they're going to show the project is achieving the outcome. And that's an issue. Yeah, that's good to know. So what we're going to do now, and I should point out that the Global Affairs Canada, their own guidelines on designing performance measurement frameworks makes this clear, that you have to have outcome indicators that reliably and validly measure the outcomes.
So they're not even following their own guidelines. This is not some thing I've cooked up myself. I'm just bringing it out into the public domain. So what we're going to do now is I will send part one and part two to the Government of Canada, Minister of International Development, as well as the performance measurement framework our summary sheet that explains all the indicators and why they have this problem.
And these two episodes, part one and two, and also sending it to the shadow critics of international development for the Bloc Québécois, the New Democratic Party, and the Conservative Party. So everybody knows this is a Canadian problem. It's not specific to a particular political affiliation. It should be a concern for all Canadians that there are $4 billion that they spend every year in developing countries. It should be properly evaluated. And we'll leave it at that.
So Benjamin, thank you very much for your time. Thank you for having me. And thank you for your expertise and insight. It was very helpful. All right, David, thank you too. Okina. Ok. All right. Bye. And have some Bangkok for me, please. In Italy. Whenever you are coming to Ghana, you know. That's the same. That's the same. All right. Midasi. Ok. Mr. Midasi. Midasi. Okina. Ok. All right, David. Bye bye. Bye bye.