Welcome to the Improving Development Evaluation Podcast. I'm your host David Wond and welcome to season two, episode one. And in this episode, we're featuring the International Development Organization United Nations Population Fund, UNFPA. You can learn more about the United Nations Population Fund at their website, unfpa.org. And we're focusing on a project that the UNFPA is delivering in Somalia and its title is Strengthening Sexual Reproductive Health and Rights Through Midwives.
And the budget for this project is 10 million Canadian dollars, courtesy of the Canadian taxpayer and funded by the government of Canada. And we're going to be reviewing their performance measurement framework. And we will start with a brief project description and then we'll move into how the outcome indicators either adequately measure the outcomes or do not adequately measure the outcomes.
This project has eight target groups and we'll list them off and then we'll get back and give you details about the services that each of the target groups receive. First target group number one are students studying midwifery. Number two are midwives after they've graduated and are practicing. Number three are students studying midwifery and practicing midwives. Target group number four are midwife associations in Somalia.
Number five are midwives who teach at midwifery schools as well as midwifery tutors. Target group number six, Ministry of Health officials in Somalia. Target group number seven, traditional birth attendance. And finally, target group number eight, Chief of Health, midwifery specialist and a family planning program manager. So let's get back to the services that these target groups actually receive due to this project costing about 10 million dollars.
So the first target group, the target group are students who are receiving midwifery training and that training includes the following. Developing a plan to hire and pay midwifery graduates. Equipment is provided to the midwifery schools. Training curriculum is developed and delivered in the midwifery schools including bringing in Canadian midwives to the classroom. Also supervision visits and technical assistance is delivered to the midwifery schools.
The midwifery schools are assessed for accreditation for midwife students to be placed for practicums. The second target group are midwives themselves. They receive continuous professional development training. Midwives are also supported to lobby to get national midwifery act into the national constitution. The third target group are students studying midwifery as well as practicing midwives. They receive online midwifery training materials that are established.
The fourth target group, midwife associations. They're trained to disseminate and develop messages through radio and social media on safe delivery and nutrition. They also receive or attend events or exchanges where I'm assuming it's Canadians go to Somalia to train members of midwife associations to quote, strengthen their leadership and management capacity including their ability to advocate for midwifery as a profession.
These midwife associations also support to advocate for zero tolerance for female genital mutilation at hospitals. They also train these midwife associations, girls at schools, universities and youth representatives on how to identify cases for referral to midwifery services. This association also develops a mobile app to reach youth to access midwifery services.
The association is also supported to train traditional birth attendants and health workers and midwives on breastfeeding and nutrition and food security. Target group number five, midwives who teach in midwifery schools and midwifery tutors. We're not clear who does it, but policy guidelines and protocols are developed and taught to the students studying midwifery by the midwives who teach them. Target group number six, the Ministry of Health officials.
Their responsibility as part of this project is to review health data to deliver 48 outreach events where women receive midwifery services. Now it's not clear what the dollars are used for, but I'm assuming they're used to pay for those outreach events, those 48 outreach events. Target group seven, traditional birth attendants. The midwifery association does outreach to them to promote and increase referrals to midwives.
Also I should add, midwifery associations deliver 48 awareness events on midwifery and train midwives on nutrition and nutrition counseling. Target group number eight, which is the chief of health, the midwifery specialist and the family planning program manager. The project funds are used to pay 25% of the salary of the chief of health, 50% of the salary of the midwifery specialist, and 20% of the family planning program manager in this project.
And their responsibility, these three individuals, is to deliver a mass media campaign, which I suspect relates to the training of midwives on message dissemination, which I was talking about earlier. So that's a brief overview of the target groups and the actual services that they receive. And as you can tell, a lot of it has to do with training. So if we look at the performance measurement framework, there are eight outcomes for this project related to those services.
And there's 13 outcome indicators that are in the performance measurement framework that attempt to measure whether or not those outcomes have been achieved. So I'll just briefly go through the eight outcomes and then I'll go back in detail, provide the outcome indicators and whether or not they adequately measure the outcomes. First outcome is project is effectively supported in terms of operations, security, communications, advocacy, monitoring, and evaluation.
Outcome number two is improve the role of midwives in increasing knowledge awareness on maternal nutrition and contributing to sexual reproductive health outcomes for target communities. Outcome number three, increased marginalized women and girls access to quality and rights-based midwifery services, including sexual reproductive health rights-based services. Number four, improved regulatory environment and association leadership for rights-based quality midwifery practice.
Number five outcome, increased quality of midwifery training and practice both in pre and in service to provide rights-based and evidence-based knowledge and service delivery, including sexual reproductive health rights. Outcome number six, increased use of skilled midwives and reproductive health services by women and adolescent girls age 14 to 49.
Outcome seven, improved availability and accessibility of professionally trained and certified midwives able to provide quality rights-based sexual reproductive health services to women aged 14 to 49. And finally, outcome number eight, reduced maternal mortality and increased sexual and reproductive health and rights for women aged 14 to 49 in Somalia.
So if we look at the first outcome, project is effectively supported in terms of operations, security, communications, advocacy, monitoring and evaluation. There is no outcome indicators at all for this and it may have something to do with it's been thrown in there just for operations to make sure that the project is going smoothly.
I will point out that one thing they should be doing, which a lot of these projects are not doing is taking a certain percentage of the budget and allocating it towards evaluation because a lot of these organizations claim there is a lack of funding for the evaluation. So you could crudely put, take 10% of the 10 million or 1 million for proper evaluation.
And in fact, you could use that for measuring the outcome indicators recommended that I'll be doing in this episode because if you've watched season one, one particular NGO that did show up to respond to our critique was saying they don't have enough resources and that is possible. But you would think if they do proper outcome indicator design that they would put their money where their mouth is and the government of Canada would make sure that there was sufficient funding within the project.
So if it's 10 million bucks, they should take a certain percent of it immediately and put it aside for evaluation and then the remainder can be used to deliver the services rather than using it all just on delivering the services. So that's the recommendation there is at least have a fixed amount. It depends of course on the project, but 10% would be a good rule that you could put aside for the evaluation, the measurement of the outcome indicators.
So outcome number two, improve the role of midwives in increasing knowledge, awareness on maternal nutrition and contributing sexual reproductive health outcomes for target communities. The outcome indicator there is community awareness engagement and training package developed and implemented. And as you can tell, that's not an adequate measure of increasing knowledge and awareness. So that is not related to the outcome.
So what they should be doing instead is they need to measure levels of awareness among the target groups, which are youth and girls that the midwife associations are sending messages to and training them to make referrals. Another option is to track the number of referrals against locations where these project midwives are not sending out messages.
So in terms of the violation of the OECD development assistance committee criteria, it's violating the effectiveness criteria, which the DAC question is, is the intervention achieving its objectives? No, it is not because the outcome indicator is not measuring levels of awareness, which the intervention is expected to increase. Also you could argue it's not addressing the impact question, which is did the interventions make a difference?
Now it's a bit debatable, but you could argue here that you could look at impact in the areas where the project is trying to increase referrals by sending out messages via social media. And you could look at areas where those messages are not being sent out and compare whether or not the referrals are lower in the areas where the project is not operating. That would be an example where you could show impact. But again, they're not even doing that here.
So that's another violation of impact using the DAC criteria. Third outcome is increased marginalized women and girls access to quality and rights-based midwifery services, including sexual reproductive health services. One of the indicators there is the number of midwives graduating in accordance to WHO standards, World Health Organization standards.
So even if they graduate up to standard, we don't know if they've got access, increased access these marginalized women and girls, unless we go to the facilities and make sure that these midwives show up and start working, right? This outcome indicator doesn't measure the outcome directly. So it's not related to the outcome. So they graduate, but they do not show up at the health facilities.
So what you would do instead is measure the number of midwives who are at the health facilities against a standard considered acceptable for adequate access. And the WHO does have standards, global standards. In fact, they're probably even country level standards where you can say there's enough health professionals per capita or per catchment area of population that's acceptable. So again, this violates the effectiveness criteria of the development assistance committee criteria on evaluation.
Is the intervention achieving its objectives? No, it is not because the outcome indicator is not measuring levels of access, which the intervention is expected to increase. Another indicator for the same outcome is the number of midwifery schools where curriculum has been updated to include rights-based approach to sexual reproductive health. So again, it's not relevant in the sense that you could have the curriculum updated, but you really want to go a step further.
So the measure is not related to the outcome. And what they want to do is if even if you improve the curriculum, it has nothing to do with monitoring increases in access, which we want to see. So again, you want to go to the health facilities instead where midwives are working to see if the number of patients served or the wait lists are considered adequate access against the agreed upon standard.
So again, it violates the effectiveness criteria of DAT, development assistance committee for evaluation. So outcome number four, improved regulatory environment and association leadership for rights-based quality midwifery practice. The first indicator there is number of midwifery associations supported for rights-based midwifery and sexual reproductive health services. Again, the measure is not related to the outcome of improving leadership skills.
So they say support it, but even if they're supported through training or whatever, what you really want to do is you want to measure the leadership skills and the skill to advocate for rights-based quality would be better. So again, it violates the DAT criteria on effectiveness because the outcome indicator is not measuring levels of skill to lead and advocate, which the intervention is expected to increase.
So the other indicator for outcome number four, number of midwifery association members supported to lead right-based quality midwifery practice efforts. Again, it's the number of associations, but it's got nothing to do with measuring the outcome of improved leadership. So again, they need to measure the leadership skills and the ability to advocate. So again, it violates the DAT criteria of effectiveness.
Then you have outcome number five, increased quality of midwifery training and practice both in pre and in service to provide rights-based and evidence-based knowledge and service delivery, including sexual reproductive health services. The indicator there is the number of midwifery tutors train in new content and evidence-based training methods or methodologies. Here they're just taking attendance of how many got trained. They're not even looking at the increased quality of the training.
And one way to do that is to look at the test scores of these tutors on their tutoring methods to see if they've improved in how to use certain methods of how to tutor. So again, it violates the effectiveness DAT criteria because the outcome indicator is not measuring levels of skill to tutor, which the intervention is expected to increase. Another indicator they've used for this outcome number five is the number of midwives trained.
So it's just a different group, not the tutors, in new midwifery content, including rights-based sexual reproductive health. So again, it's not just the number of midwives showing up to get the training. That's just taking attendance. That's not good enough. You have to show how you're going to increase the quality of these midwives who have been trained.
So again, what you need to do instead is look at the test scores of midwives on their knowledge of rights-based sexual reproductive health content. So it doesn't measure effectiveness because they're not measuring levels of knowledge on rights-based sexual reproductive health, which the intervention, the training, is expected to increase or achieve. Outcome number six, increased use of skilled midwives in reproductive health services by women and adolescent girls aged 14 to 49.
One of the indicators there is number of people provided with modern contraception by method. This is a great measure of expected outcome because the skilled midwives in reproductive health services delivered would include contraceptives. Now, we don't know if they're going to actually use them, but at least if they go to the health centers and they can show this, it's measured quarterly.
We can see that if it's going up that the, we're assuming when they graduate from these midwifery schools, they show up at these facilities and they start handing out these contraceptives. So hopefully it will show that there has been increased use and this is a good measure of that. So this is a good example, but the challenge here is we haven't got the data and it's not on the Project Browser Global Affairs Canada website.
So even though it's a good measure, the next step is we need to see the data to show that it's actually going up, the number of people receiving them. Another indicator that's very good for this outcome number six is the percentage increase in births per facility from the baseline determined at the project start. This again is an excellent indicator of that achievement of that outcome and it's measured annually.
But again, we don't know that percentage increase and we need to see that data on the Project Browser. And that would be helpful, but it's a great indicator. Outcome number seven is improved availability and accessibility of professionally trained and certified midwives able to provide quality rights-based sexual reproductive health services to women aged 14 to 49. One of the indicators for that outcome is number of midwives graduating in accordance with WHO standards.
Accessibility of course is not enough to show improved availability and access to services, right? We have to make sure they show up at the health facilities. So we have to measure the number of midwives who are at the health facilities against a standard considered acceptable for adequate access. Another indicator for this improved availability and accessibility to services outcome number seven is number of facilities with all the signal functions to provide skilled deliveries.
I'm not sure what that means, but people in the field of this area of expertise probably know what it means. It's great. It's a good indicator. We also want to know the percentage of the facilities where all the signal functions are there to provide skilled deliveries. But again, we would like to know the number and the percentage, the actual data. So this is a good indicator. It would just be ideal if it was available to the public.
The other indicator on outcome number seven is percent of respondents over the baseline reporting quality service. Now this is interesting. It's a great indicator, but what they're asking is quality. What do you think of the quality of the service? They're not asking about access or quantity. They want to know about the quality. So it's a great indicator. They just have to reword it. So instead of improved availability and access, they should change it to improve quality of services delivered.
And that's an excellent indicator. So these are some examples of where some of the indicators are actually properly measuring the outcomes, which is great. And finally, outcome number eight, reduced maternal mortality and increased sexual and reproductive health and rights for women aged 14 to 49 in Somalia. And the indicator they have there is number of women and girls provided with access to sexual and reproductive health services, including modern methods of contraception.
Well, as you've noticed, some of the previous outcome indicators already cover access to sexual and reproductive health services. So we don't need to worry about that. They need to measure the maternal mortality rate. And that's not in there, which is interesting. And it should be because that's critical because in the end, they're claiming with that outcome, they're claiming they're going to achieve it.
They're going to reduce maternal mortality, but they haven't measured the maternal mortality rate. All they're measuring is access. So in that case, it's failing in terms of the DAC criteria, you could argue impact or you could say effectiveness. Has the maternal mortality rate, thanks to the project, actually gone down? And to show that you would need to show where the project is not operating if the maternal mortality rate is higher.
Now to be fair, they have targeted all of the midwifery schools in Somalia. So you could argue it's blanketed. They've blanketed the entire country. There's no comparison group or control group where they haven't done the training and all those interventions that I described in the project. For some of them, you could argue like where they're trying to increase levels of knowledge and awareness on nutrition or breastfeeding, et cetera. And they're sending targeted messages in the community.
There could be communities where they're not sending those messages. Then you could compare the two groups and see if levels of awareness is lower where the project is not operating. But definitely they need to measure maternal mortality rate. That is quite important. So here you could argue either effectiveness is being violated or when it comes to raising levels of awareness, they're not looking at the differences or impact, which is how the DAC criteria is described.
What difference does the intervention make? So you could argue, well, maternal mortality rate has to go down. That's a difference. And also the knowledge and levels have to go up. So there's a summary of the project. And what I'm going to do now is send the performance measurement framework along with this analysis of the outcome indicators, send it to the minister for international development, as well as the shadow critics.
And also if any of you would like any of the performance measurement frameworks and the Excel summary of the outcome indicators that I've just described in this episode, I'm happy to email those to you. You can request them by email and send an email to evaluatecanadaaid.gmail.com. I'll put it in the episode notes. Thank you for listening and stay tuned for episode two, which hopefully will come a little quicker. Thanks for now.