In this episode we're going to introduce you to an international development organization. That international development organization is the United Nations Population Fund. You can learn more about the United Nations Population Fund if you go to their website at www.unfpa.org. Their headquarters is located in New York, USA.
The project we're going to be talking about, that the UNFPA has been funded by the Government of Canada to deliver, is called Human Resources for Sexual Reproductive Health Rights and Gender-Based Violence. This project is being delivered in South Sudan and the cost of this project to the Canadian taxpayer is $37 million.
And in this part one I'm going to give you a brief description of the services delivered in this project and a critique of the United Nations Population Fund, South Sudan's Performance Measurement Framework. The UNFPA have developed this performance measurement framework to measure the project's performance using the outcome indicators in that performance measurement framework. I obtained this performance measurement framework through an access to information request.
I also obtained the United Nations Population Fund Evaluation Plan for this project. However, this evaluation plan provides no additional detail as to how the outcome indicators will be used to measure the project outcomes for this project. Therefore I will not be discussing the evaluation plan as part of this episode.
But before I describe the project services that are being delivered for this project by the UNFPA in South Sudan as well as the expected outcomes to be achieved from the delivery of those services, I'm going to go to my summary and conclude that in this performance measurement framework for this project there are 16 outcome indicators that are measuring 7 outcomes for this project.
My review of all 16 outcome indicators has led me to conclude that all of these indicators, except one, are not properly measuring the outcomes. So, the organization, the United Nations Population Fund, South Sudan, country office, cannot make the claim that it's achieving its project outcomes with the services they're delivering in the project simply because their outcome indicators are not properly measuring the outcomes for this project.
As you probably know from the trailer for this podcast, those 16 outcome indicators fall into one of five problem areas. In this project, eight of those 16 outcome indicators fall into the no comparison group problem, which is where there's no comparison group of people outside of the project that are also measured on the same outcome.
Which means even if the people in the project are achieving the outcome, we don't know if they're achieving them any better than another group of people that could also achieve the outcome outside of the project without receiving the project services. Another seven of the outcome indicators out of the 16 fall into the problem area where the outcome indicator is not measuring the outcome at all and is measuring something not related to the outcome.
So, and finally, we have one outcome indicator that is properly measuring the outcome. So, before I give you three examples where the outcome indicator fails to properly measure the project outcome, I'm going to briefly describe the services the project delivers and the target groups that receive those services. And then I'm going to list the seven outcomes that the project expects to achieve from delivering those services to those target groups. There are 11 target groups in this project.
The first target group is staff at targeted health facilities and the services being delivered to them are training on a maternal and perinatal death surveillance and response system so that these health facilities have a functional system to report on maternal and perinatal deaths. Staff at these health facilities are also provided training on how to repair and maintain solar and water supply systems since these health facilities are provided essential medicines, solar panels and water supply.
These health facilities are also provided with health care workers, especially health care workers trained in delivering gender responsive sexual reproductive health services and being able to respect the rights of those receiving these services. A second target group are health care personnel. This group receives training in how to mentor and supervise students completing their practicums.
A third target group are staff at one stop center family protection centers who are provided training on how to deliver information and services for victims of gender based violence, including how to take cases to court. A fourth target group are women and girls and men and boys who are delivered messages on the importance and availability of using sexual reproductive health and gender based violence services.
A fifth target group are community action groups and religious and civic leaders that receive messages to promote women's use of sexual reproductive health and gender based violence services. It is not clear from the performance measurement framework, but we can assume that either training or messages are also delivered to these community action groups on how to prevent and respond to gender based violence and child marriage.
Because the outcome in the PMF is and I quote, community structures are mobilized to take actions to prevent and respond to gender based violence and harmful practices. A sixth target group are men and boys who are oriented on their role as men and boys to prevent and respond to gender based violence and other harmful practices, as well as promote women and girls rights to access sexual reproductive health and gender based violence services.
A seventh target group are service providers and community based organizations, state and national coordination bodies who receive training on sexual reproductive health and gender based violence prevention and response. An eighth target group are health training institutions, including tutors within these institutions where these tutors are trained on how to train health professionals in gender sensitive sexual reproductive health rights and gender based violence services.
These institutions are also provided with educational materials and solar power. These health training institutions include health sciences institutes that receive quality assurance regular field supervision visits. A ninth target group are health professional associations that are provided with technical support on how to advocate for and monitor sexual reproductive health and gender based violence services.
A tenth target group are Ministry of Health staff at national and state level who are provided with technical support on how to coordinate and manage sexual reproductive health and gender based violence services. Finally, the Performance Measurement Framework, PMF states an expected outcome is of laws and policies being developed, revised and enacted with the Minister of Health and Parliament passing those acts. Those acts are the Midwife Act and the Anti-Gender Based Violence Act.
Therefore the 11th target group would be the Minister of Health and Parliament. However, the PMF does not indicate if the Minister of Health and Members of Parliament need training to achieve this expected outcome of passing these acts. They probably are assuming in this project that the Ministry of Health staff and the Health Professional Association staff, not the politicians, through their training received from this project is sufficient to convince these politicians to pass these acts.
So that gives you an idea of the services this $37 million project delivers in South Sudan by the United Nations Population Fund. Now after those services are delivered they expect to achieve for these target groups seven outcomes. And those outcomes are, outcome number one, improved national and state government capacities to develop and implement sexual reproductive health, gender based violence, HIV policies, legislation and coordination mechanisms.
Outcome number two, strengthened capacities of health training institutions to produce quality human resources for sexual reproductive health rights, gender based violence and HIV services. Outcome number three, increased awareness among women and girls to utilize gender responsive sexual reproductive health, gender based violence and HIV services.
Outcome number four, strengthened capacity of targeted national and state level health facilities to provide quality integrated sexual reproductive health rights, gender based violence and HIV services. Outcome number five, strengthened enabling environment at the national state and local level for women and girls access to sexual reproductive health rights and gender based violence services.
Outcome number six, increased use of gender responsive sexual reproductive health rights and sexual gender based violence services by women and girls, especially those living in vulnerable situations. And finally, outcome number seven, increased realization of sexual and reproductive health and rights of women and girls in South Sudan. Now we're going to look at three outcomes and three outcome indicators for each of those outcomes and explain how they do not properly measure the outcome.
So let's start with outcome number five, strengthened enabling environment at the national state and local level for women and girls access to sexual reproductive health rights and gender based violence services. One of the two outcome indicators in the PMF that is used to measure strengthened enabling environment is proportion of targeted health science institutions that meet the minimum educational standard for training health professionals.
We're going to assume that if these health science institutions are training health professionals up to a minimum educational standard, that this is good for women and girls who receive sexual reproductive health and gender based violence services from these students who graduate from these institutions and start delivering these services to these women and girls. So this is the same as saying we have a strengthened enabling environment for women and girls when they access these services.
That seems to be a good match between the outcome indicator and the outcome statement. But when we look at the performance measurement framework for this outcome indicator, we see that an assessment is done every three months at the institution along with a progress report.
Upon reviewing the performance measurement framework as well as the evaluation plan, I cannot find a measurement tool on how this proportion of health science institutions that meet this minimum educational standard for training health professionals is calculated. The PMF shows that the people inside of this institution who are responsible for training health professionals will be assessed as to whether they meet this minimum educational standard.
The outcome indicator is excellent, but there is no explanation in the performance measurement framework as to how they calculate this percentage. We need to ask the United Nations Population Fund, South Sudan representative, where is the method and calculation that comes up with this percentage? Also in the PMF, the staff at the health science institution itself, the staff at the Ministry of Health and the United Nations population staff are responsible for coming up with this percentage.
It would be better if an external third party would be responsible for assessing these staff in these health science institutions to see what percentage of them have reached this minimum educational standard for training health professionals. Then this could be repeated across all health science institutions to get the proportion of all health science institutions that achieve this minimum educational standard for the indicator.
That would remove any bias since right now the United Nations Population Fund staff and these health science institution staff are assessing themselves on how well they train the health professionals to reach this minimum educational standard.
You may recall from the project services I described earlier that tutors at these health science institutions are trained by the project on how to train health professionals to be quote gender sensitive in their delivery of sexual reproductive health rights and gender based violence services.
The project also delivers quality assurance regular field supervision visits at these health science institutions which should or could include supervision of these tutors in how they train midwives in delivering gender sensitive sexual reproductive health rights and gender based violence services.
So what is the minimum percentage of tutors at these health science institutions that first need to reach this minimum educational standard based on this assessment that's listed in the PMF before that health science institution as a whole is declared to have reached this minimum educational standard for training health professionals.
And then we have to figure that out and then we can move on to the next health science institution and repeat it so that for all health institutions we figure out what the percentages that meet this minimum educational standard. In the performance measurement framework there is a target where they're trying to achieve 80% of health science institutions reaching this minimum educational standard.
But if only one tutor let's say out of a hundred tutors in one of these institutions shows up for the assessment it passes it. Is that enough to say that there's a sufficient number of tutors that have passed the minimum requirements to train the midwives before we move on to the next health science institution?
So the percentage could be misleading we just don't know what's going on inside the institution to come up with this magical percent of health science institutions that have reached this minimum educational standard. So again we need to ask the United Nations population fund what percent of tutors do you require to assess before you conclude that the whole health science institution of all the tutors is good enough and it's reached the minimum educational standard.
And then what percent of those tutors do you have as a minimum requirement to say ah we got 50% of them met the assessment minimum educational standard therefore we conclude that the whole health science institution is good enough we'll go on to the next health science institution before we get to this target of 80%.
So we need to ask United Nations population fund about that and what's the assessment tool that they actually use to say oh that tutor you're up you're ready you have reached the minimum educational standard when you're training midwives on delivering gender-based violence and sexual reproductive health services you're okay and then they move on to the next tutor and assess.
So we see there's progress reports we see that they do an assessment but there's no mention of how they come up with this percentage and that should be included in the performance measurement framework and even in the evaluation plan which is there's nothing in there it's just a generic evaluation plan for any project it's not specific to this project so that is a problem.
Let's look at outcome number six increased use of gender responsive sexual reproductive health rights and sexual gender based violence services by women and girls especially those living in vulnerable situations. One of the outcome indicators measuring this increased use of gender responsive sexual reproductive health rights and gender based violence services is proportion of women aged 15 to 49 who receive four or more antenatal visits as per targeted locations.
This indicator has two problems first there is no mention of the proportion of women who receive four or more antenatal visits where those antenatal visits were specific to gender responsive sexual reproductive health rights as reported by the women and girls who receive these antenatal services. Remember this project trained tutors at health institutions to quote train health professionals in quote gender sensitive sexual reproductive health rights and gender based violence services.
The indicator should include the same statement in the outcome on gender responsive sexual health reproductive rights to read proportion of women aged 15 to 49 who receive four or more antenatal visits as per targeted locations and where those visits were reported by these women as being quote gender responsive and respecting their sexual reproductive health rights.
So we need to ask United Nations population fund why did they drop the increased use of gender responsive sexual reproductive health rights from the outcome indicator when it is expected to be achieved as stated in the outcome. The second problem is that the outcome indicator is only measured in quote targeted locations.
So even if the proportion of women who receive four or more antenatal visits and who report those antenatal rights as being gender responsive and respecting their sexual reproductive health rights is going up thanks to the projects men and boys promoting women and girls right to access those services and the projects training of tutors to train health professionals on being gender responsive and respecting sexual reproductive health rights.
It could be going up even more in another part of the country where the project is not delivering those training services right. So we need a comparison group of women not located in the project area that are also receiving antenatal services to also ask them if the antenatal services they received also respected their sexual reproductive health rights and were gender responsive. Then we compare their percentage against the percentage of women in the project who report those antenatal services.
We would expect the project percent to be higher. Finally, outcome number seven increased realization of sexual and reproductive health and rights of women and girls in South Sudan. One of the outcome indicators for this outcome is the maternal mortality ratio. This indicator is a universally agreed upon indicator and could be seen as a proper measure of increased realization of sexual reproductive health for women and girls.
If these women and girls are experiencing lower rates of death due to pregnancy after receiving sexual reproductive health services in this project. But there are two points I need to make clear here. First, the project delivers these necessary sexual reproductive health services to women and girls in quote targeted locations. For example, from the performance measurement framework, health facilities in these targeted locations are provided with health care workers.
This suggests there are health facilities outside of these targeted locations that do not receive these health care workers along with the sexual reproductive health services they deliver.
Secondly, this project also delivers other services other than sexual reproductive health services to women and girls such as training for tutors and mentors of midwife students on how to deliver gender responsive sexual reproductive health services and how to respect the sexual reproductive health rights of women and girls when delivering these services. There's also training to men and boys to promote sexual reproductive health rights for women and girls.
There's also training to Ministry of Health staff on how to manage the delivery of sexual reproductive health services to these women and girls. There's also training to health profession associations on how to advocate for and monitor sexual reproductive health services to these women and girls.
So even if the percent of deaths due to pregnancy goes down, we need to show that this $37 million project did it better with greater reductions in maternal deaths than in locations where this project was not operating. And we also need to show that all these trainings delivered to all these other target groups worked better than these same target groups in locations where the project was not delivering these trainings. Once again, we need comparison groups for each of these target groups.
So now what I'm going to do is email this episode, part one, to the United Nations Population Fund representatives, inviting them to attend part two of this podcast where they can respond to this critique of this performance measurement framework. I will also send to the UNFPA representatives this performance measurement framework as well as my summary critique of all of the 16 outcome indicators for the project.
And I'm also going to give the United Nations Population Fund representatives the option of sending me a written response to my critique if they are unable to attend the podcast. An evaluation expert will be attending part two of this podcast. And in part two of this podcast, we will be covering all of the seven outcomes for the project with at least one outcome indicator for each of the seven outcomes.
If you have any comments or suggestions for this episode or wish to receive a copy of the performance measurement framework and my summary critique of all 16 outcome indicators for this UNFPA project, you can email me at evaluatecanadaaid at gmail.com.
And if there are any evaluation experts who'd like to be on the podcast to look at other performance measurement frameworks, you can email me and I'd be happy to send you the remaining performance measurement frameworks that we have lined up for season one. Thank you for listening and stay tuned for episode three, part two, featuring the United Nations Population Fund.
And there's $37 million project in South Sudan called the Human Resources for Sexual Reproductive Health Rights and Gender-Based Violence.