Welcome to the Improving Development Evaluation podcast. I'm your host David Wand and welcome to season two, episode two, where we're going to feature the international development organization called Doctors of the World. And if I recall correctly, they are located in Montreal, Canada. They have received $14,886,968 according to the Government of Canada Global Affairs Canada project browser website. For a project they are delivering in the West African country of Burkina Faso.
And the name of that project is called Access to Health Rights in Northern Burkina Faso. And you can learn more about Doctors of the World, the international development organization at their website www.medecinsdumonde.ca. And in this project of $14 million, courtesy of the Canadian taxpayer, there were eight expected outcomes to be achieved from delivering services in Burkina Faso for that cost of $14 million.
And for those eight outcomes, they had 15 outcome indicators that they used from the performance measurement framework to see whether or not those eight expected outcomes had been achieved.
My review of those 15 outcome indicators indicated that 13 out of the 15 were not valid measures of the expected outcomes and therefore Doctors of the World cannot make the claim that even if those outcomes were achieved, they cannot claim that the project costing $14 million was responsible for the achievement of those expected outcomes.
Before I go into detail about those outcome indicators and the outcomes, I'm going to give a brief description of what the project from the performance measurement framework actually delivered. And at any time, if you're interested in getting the performance measurement framework plus the critique in an Excel sheet that I have, you can email me at evaluatecanadaaid.gmail.com and I would be happy to email those documents to you. So let's begin with a brief description of the project.
The project in Burkina Faso delivered by Doctors of the World has roughly 16 target groups and each one of those target groups receives a certain amount of services. And with those services, they expect some of those target groups to achieve certain expected outcomes. So starting with target group number one, International Dialogue Forum participants. They are supposed to receive from Doctors of the World presentations at these international forums to advocate for access to health.
Second target group, authorities and stakeholders in Burkina Faso at local and national level and they are to receive from Doctors of the World presentations to this target group to advocate for access to health. Target group number three are community and religious leaders and they are to receive support in quote, facilitating awareness sessions. Target group number four, peer educators.
They're supposed to receive trained sessions to promote to men and boys their responsibility on their right to health, sexual reproductive health and gender based violence. Target group number five, men and boys. They are receiving training on the right to health, sexual reproductive health and fighting gender based violence in quote, targeted areas. Target group number six, civil society organizations.
They are trained on how to promote health rights, sexual reproductive health and fighting gender based violence. Target group number seven are women and their allies and they're trained and mobilized for the right to health, sexual reproductive health and the fight against gender based violence. Target group number eight are women and girls and they are sensitized and supported on health and gender based violence in quote, targeted areas.
Target group number nine are families with children under five and in targeted areas they are trained on the mid upper arm circumference measure to enable them to detect nutritional edema and make referrals. Target group 10 are communities, women, men and adolescents in these communities. They are to be sensitized on primary health care services, sexual reproductive health and gender based violence.
Target group 11 are community based health agents and community relays and they are to be trained on prevention, care and referral for primary health care, sexual reproductive health and gender based violence. Target group number 12, it's not clear who exactly they are but these are people that are accepting referrals to treat women and girls and they're trained on referrals treated with no financial charge to patients.
So I guess what that means is they're treating patients that they've received referrals from women and girls where they're not going to be charging them for services. Target group number 13 are health providers. They're trained in epidemic management and rapid response during a humanitarian emergency. Target group number 14 is remote areas and this is where mobile clinics go to those remote areas deployed to deliver free medical services including sexual reproductive health services.
Target group 15 are supervision teams and they're trained on quote provider supervision techniques and finally target group 16, I don't know what the acronym stands for FOSA, FOSA. They are supervised and supported to deliver primary health care and treat gender based violence cases. So that gives you an idea of the target groups and the services delivered at a cost of roughly $14 million.
So now we're going to look at the eight outcomes and for those eight outcomes the 15 indicators that were used to measure whether or not those eight outcomes were achieved and there could even be more but from my review of the performance measurement framework this is what I figured out were the eight expected outcomes. The first one was and I quote increased mobilization of actors to take into account gender inequalities in particular sexual reproductive health issues in emergency response.
The outcome indicator number one, actors involved is measured enough and also there's no comparison or control group outside of the targeted areas for this project so we don't know if the number of actors outside of the project also were involved in issues of gender inequality and access to health in the response emergency.
So keeping with the trailer for season two this would violate the development assistance committee criteria on impact because there's no comparison group or control group so they can't make a claim that any difference they see in this indicator such as an increase in the number of actors it could happen anyways regardless of the project so it violates the impact criteria for the development assistance committee and this is a big thing in international
development where you'll see they often go to these five questions or issues of evaluation in the development assistance committee criteria one relevance two coherence three effectiveness four efficiency and five impact and six sustainability.
So in this performance measurement framework in this project most of the violations are on impact they just simply can't claim that the project is responsible for achieving the outcomes so indicator number two for this same outcome is the number of advocacy priorities taking into account gender inequalities in particular sexual reproductive health issues in the emergency response presented to the authorities for consideration in local and
national policies again problem is it's not measured frequently enough and there's no comparison group if we look at the performance measurement framework I can't see it right here I'm claiming it's not measured very frequently it's probably measured only once every it's measured up every six months the first one that we just covered so anything could happen in six months between those measures actors in other communities outside of the project
could also be getting mobilized and it's the same for this indicator here indicator number two they only measuring it every six months and the other thing that's interesting here and it's it's been done before is presented to the authorities for consideration in local and national policies a policy is just a piece of paper what you could do instead is just see if expenditures addressing gender gender inequalities has gone up actually expenditures
not the budget that they claim they're going to spend but after what have they actually follow the money and see if it's been spent so for example did they spend money to increase enrollment of girls in primary schools or did the number of girls or women with access to sexual reproductive health services the number of visits free of charge actually go up thanks to maybe an increase in expenditures in that area by the government so that would
be better measures rather than this advocacy priority taking into account gender inequalities to see if after the advocacy did they actually result in increased expenditures and again you could still argue even if it goes up the expenditures it may have nothing to do with the project that's why you need to have some sort of comparison group or control group the next outcome is called increased co-responsibility of men and adolescents in favor of access
for all to the right to health in the targeted areas and the outcome indicator for this outcome this is outcome indicator number three for outcome number two proportion of men and adolescents who participated in project activities who report having adopted behaviors in favor of equitable gender norms obviously there's self-reporting bias there they're going to say sure sure I adopted the right behaviors because it's hard to measure the behaviors directly so
self-reporting is going to have a bias so even more importantly here you need another comparison group outside of the project where men and boys weren't being engaged with to change their attitudes right so that is what you need to do here and again this violates the impact criteria the DAC development existence committee criteria because we don't have a control group we can't claim that the men and boys outside of the quote targeted areas
also had a preferable shift in attitude we just don't know so we need to measure men and boys in areas outside of where the project is going on and this is a challenge of course but it should be at least attempted so again for that increased co-responsibility outcome there's another indicator they use outcome indicator number four proportion of men and adolescents participating in activities that have quote increased their knowledge of the
right to health including sexual reproductive health and the fight against gender-based violence now here you're measuring knowledge levels which is fine but again what you need to do is have a comparison group because their knowledge levels could go up anyways in other areas where they're not operating the project so again this violates the impact criteria for the development assistance committee criteria on evaluation the next outcome number three
is strengthened capacities of women and adolescent girls affected by the conflict to exercise their right to health in particular with regard to sexual reproductive health in the targeted areas and the outcome indicator number five for that is proportion of women and adolescent girls participating in the empowerment program who have quote increased their knowledge of sexual reproductive health so again if you want to show an increase in knowledge you
have to show due to the project so ideally what you'd like to do is find a comparison group of women and adolescent girls outside of the project and see how their knowledge levels are going before during and after the project and then you could show again that the project is having some statistically significant one-tail positive effect right just to find some women and girls not in the project areas another option is if you don't want to use
control groups you could just measure more frequently and just show that the knowledge levels are going up rather than just a pre and a post test level I again I'd have to look at the performance measurement framework in detail to see how frequently they measure this another outcome indicator for this strength and capacity of women and girls specific to their knowledge levels is quote number of new initiatives in favor of sexual reproductive
health or the right to health taken by influential women and their allies so this is a separate group of women who take it up and again there's no comparison group here so this violates the impact criteria and here I have made a note according to the performance measurement framework this is only measured once a year so we could experience this increase anyways in between the time that the women and girls are trained on their capacity for their right
to health in targeted areas and whether or not these new initiatives have been taken up by influential women and their allies so it's only measured once a year anyways so even without the control group once a year is not frequent enough outcome number four women and adolescent girls sensitized and supported in terms of health and gender-based violence in targeted areas and outcome indicator number seven for that outcome number four
is proportion of women and adolescent girls in the community who have completed the health sexual reproductive health and gender-based violence empowerment program in targeted areas that's just attendance showing up but that doesn't indicate how many quote got sensitized they showed up for the program but did they get sensitized whatever that means from receiving that empowerment program so it's just attendance so what they need to do is find a better indicator
to actually measure that sensitization due to the empowerment program that they have received and I also looked at the performance measurement framework and noticed the annual empowerment program activity reports once a year so again it's not very frequent if they want to show that their sensitivity levels have gone up or their empowerment levels have gone up they have to measure more frequently than just once a year and then get a better
indicator rather than just showing up attendance who completed the program which is what they currently have and also maybe have a comparison group if they can here is it's a good example of how the indicator fails to validly measure the outcome in the first place where they're just taking attendance rather than measuring any levels of sensitivity or empowerment so outcome number five community and religious leaders quote strengthened and mobilized for
the promotion of the right to health for all including sexual reproductive health including the fight against gender-based violence this was initially reported as an output but strengthening suggests the ability to advocate and promote so I considered it an outcome the outcome indicator for this is the number of community and religious leaders supported in facilitating awareness sessions this would suggest just attendance they're not really looking at their
ability to mobilize their ability to advocate and their ability to promote so again the indicator fails to measure any increases in the ability to promote health rights for all it only shows the number who were supporting in facilitating awareness sessions but no measure showing that their ability to promote health rights for all has been strengthened so they should introduce a measure that shows that those who attended the community and
religious leaders who delivered the awareness sessions experienced increases in awareness preferably an indicator and measurement tool that has been shown to be a valid measure of the ability to promote health rights for all and increase awareness plus measuring those who attended on their awareness levels against a non-project target group so outcome number six increased capacity of conflict affected communities especially women and
adolescent girls to utilize primary health care services including sexual reproductive health outcome indicator nine for this outcome number six proportion of women men and adolescents targeted by the project who can name at least three types of modern contraceptive methods again this is a knowledge indicator so if you want to show that the project services targeting these particular individuals has increased their knowledge on contraception
you need to have a comparison group or a control group so this again violates the impact criteria for the development assistance committee criteria on evaluation the performance measurement framework shows that the percent is measured at the start middle and end of the project but no mention of a controller comparison group so with a control group and the statistical sampling they mention in the performance measurement framework they actually mention it they could
complete if we follow their interest in statistics a one-tailed hypothesis test and see with 95 percent confidence if they can claim that the percent increases were statistically significant and due to the project training so that's what they have to do otherwise right now it violates the DAC criteria on impact because they're not showing that their project group is statistically significantly greater in terms of the percentages who can name at least
three types of modern contraceptive methods another outcome indicator for that outcome number six this is outcome number indicator number ten proportion of women men and adolescents quote who have improved their knowledge during training and awareness raising on the use of primary health care services including sexual reproductive health same issue no comparison or control group and they according to the performance measurement framework they only
measure this once before the training and after the training so that could show an increase due to the training but if we don't have a control group or a comparison group you could show also another group of women who didn't show up for the training somewhere else in the country who also just over time improved increased their knowledge levels and awareness levels so that has to be taken into account and once again that violates the DAC development
assistance committee criteria on impact evaluation outcome number seven improving the social community and political environment in favor of the right to health in particular sexual and reproductive rights and gender equality for women and adolescent girls affected by the conflict in northern Burkina Faso outcome indicator 11 for this outcome number seven proportion of women men and adolescents targeted by the project having improved their behavior
in favor of sexual and reproductive rights this is tricky because I'm assuming having improved their behavior this has to be self-reporting which is biased and again no control group or comparison group outside of the project so even if these percentages increase you need a control group where these women men and boys have not received the various trainings etc from this project their behaviors could also increase so the DAC impact criteria is
not met here also same in outcome number seven we have another outcome indicator number 12 number of sexual reproductive health initiatives undertaken by civil society organizations that seek to mitigate traditional practices harmful to women and adolescent girls according to the performance measurement framework this is only measured once per year and again there's no control group so even if you don't have a control group or a comparison group outside
of the project you could measure it more frequently and that might help in allaying concerns that these initiatives would happen anyways regardless of the trainings that these women men and adolescents or CSO civil society organizations receive to uptake these initiatives so that's the other problem there again no control group no comparison group so the DAC criteria of impact cannot be claimed to be achieved another indicator for this same outcome outcome indicator
number 13 number of national policies or guidelines adopted or revised taking into account gender inequalities in particular sexual reproductive health issues in the emergency response outcome indicator third now again you could argue here that the number of national policies adopted or revised is really not related to the outcome of improving the political environment in favor of the right to health in particular the right to access to sexual reproductive
health services because as I mentioned earlier a policy is just a piece of paper a better measure would be an actual increase in the amount and percentage of the government budget that was actually spent on women and girls specific to accessing sexual reproductive health and gender based violence services so even though they have a policy on a piece of paper do they actually put their money where their policy is this is a common sort
of indicator that many governance organizations when monitoring governance and government institutional performance look at because they can have all the policies in the world but really what you need to do is follow the money especially when it comes to in this project they are doing a large portion of this 14 million dollars in advocacy so really what you should be doing is tracking government expenditures outcome number eight increased
enjoyment of rights relating to health in particular sexual and reproductive health of women men girls and boys affected by the conflicts in northern Burkina Faso here we have two indicators and I look at these indicators and I think they're fairly universal in this area and the first indicator outcome indicator number 14 of the project is percentage age 15 to 19 who have given birth to their first child who or who are pregnant with their first
child and the idea clearly there is to try to reduce this percentage and I think it's a universal global indicator that's used which is fine the indicator is a valid measure of the outcome of increased access and enjoyment of rights relating to health and what is tricky here is this is the only indicator where you could argue it is a valid measure but again all of the services I described for those 16 target groups that this 14 million dollars
is paying for in the delivery of services to those target groups and you're hoping that this indicator will go down that's fine but to show that you need to have some sort of comparison group or ideally in the demographic health survey they call it where they measure this one of the questions on that demographic health survey should be by the way did you participate in any way shape or form in the doctors of the world project paid for by global
affairs Canada and right there you could do a split and see without getting too methodologically complex here because I don't claim I'm a statistician happy to get email saying sorry that's not how you do it but at least we could compare the two groups when we do the survey and those who say no what was their rate of pregnancy between age 15 to 19 compared to the group that said yeah I did show up at those awareness sessions on contraception and then you could
do a comparison but right now I don't see that in the performance measurement framework it's a great indicator but we don't know if the project for 14 million dollars actually contributed to a reduction in that percentage age 15 to 19 that reported being pregnant we just don't know and finally the last outcome indicator number 15 proportion of women who have decided to use family planning alone or jointly with their husband or partner again
an excellent indicator that suggests if this project is successful and has an impact quote unquote this indicator would be going up the proportion of women who have decided to use family planning would be going up and it would be going up statistically significantly compared to a group of women who were not participating in the project so ideally what you want to know if this question is being asked in the demographic health survey did they participate
in this doctors of the world Burkina Faso project in any way and then you could see if the project was in any way responsible for contributing to this increased improvement and performance on that indicator so that's my summary and as I'll repeat I would say in its current form right now the performance measurement framework is flawed and the project and doctors of the world cannot make the claim that their 14 million dollar project is achieving
the outcomes that they claim they're achieving in the project so now I'm going to send this episode to the Minister for International Development along with the shadow critics and I'm also going to invite doctors of the world to provide a written response to this critique and also an invitation if they wish to be on the podcast to respond to the critique and if that happens we'll be doing another episode thank you for listening and stay tuned
for a further episode and I do have some good news that we have located in the United States the actual grant for an American USAID project that was funded to an implementing partner and so hopefully that will in turn result in getting the activity monitoring evaluation learning plan and then we can critique that which is similar to these Canadian performance measurement frameworks bye for now