Welcome to the Improving Development Evaluation Podcast. I'm your host David Wand and in this trailer we're going to give you some more detail about what the podcast is actually trying to do. We're going to talk about three things. One of the content of each episode including who the guests will be on the
podcast. Secondly we're going to talk about season one about what we plan to cover in season one which includes 12 episodes and then finally we're going to talk about some ideas about how you the listener may want to get more involved with the podcast in addition to just hopefully listening to it. So let's get right into it. So the content of each episode we're going to have two parts.
In part one the host is going to introduce a new international development organization for each episode and in this part one I'm going to talk about the project that this new international development organization has been funded to deliver in a developing country by the government of
Canada. So I'm going to briefly describe the services the project delivers, the cost of the project to the Canadian taxpayer and most provocatively and most importantly I'm going to conclude at the end of part one that we've got a problem. That is the project has failed to achieve its outcomes and you may be wondering how I'm going to get there. First of all what do we mean by outcomes? So for example a project trains people on how to
grow peanuts and how to sell peanuts. The outcomes you would expect from that training would be that those people would learn how to grow peanuts by having increased skill and knowledge levels as well as increased skill and knowledge levels on how to sell those peanuts. And I've come to the conclusion that they have not been able to in the project claim that those outcomes have been achieved.
Why? Well it's because I've acquired through a access to information request a PMF, a performance measurement framework. These are not available to the public. You have to first make an access to information request and I have acquired 12 of them. So in part one what I'm going to do is briefly describe how the PMF or performance measurement framework fails
to adequately measure these outcomes. So they fall into five problem areas and the first problem is we have to look at the outcome indicators that are trying to measure the outcomes like I've just given an example. So in the example and the first problem would be people show up for the training on how to grow peanuts and how to sell those peanuts but they're never measured on their skill and knowledge levels. Instead they just take attendance and
everybody goes home. So that's the first problem. The second problem they could have is they show up they measure their skill levels but they don't measure them directly. Instead they ask them how do they feel about their ability to grow peanuts and their ability to sell those peanuts after getting the training on how to grow peanuts and how to sell peanuts. They're asking them how they feel about their ability. They're not being measured objectively like through a test
or a certification process or whatever. So that's the second problem. The third problem is sometimes the outcome indicator doesn't measure the outcome at all. So let's say expected outcome from the training on how to grow and sell peanuts is an increase in income from selling those peanuts. Sounds logical. Problem is they don't say show me the money. Let's see your bank account. Let's see your bank balance from selling those peanuts. Instead what they do is they say show me a savings plan
that you have. Well that's not really related. They've got to show direct measurement of the income. So that's the third problem. The fourth problem is maybe they show up they properly measure them objectively on their skill and knowledge levels on how to sell peanuts and how to grow peanuts. But they only do it once before the training and they do it once
after the training. So this is a problem also because let's say they measure them as a group on their skill and knowledge levels on how to grow peanuts and how to sell peanuts before the training starts. And then they get their test scores back and some of them say oh jeez I'm really bad. My knowledge levels are really low on how to grow peanuts and how to sell peanuts. I think I'm going to go down the road and have a chat with a very successful peanut farmer and ask him or her
how do you do it. Or maybe they'll go online and figure it out if they have access to the internet through a YouTube video on how to grow and sell peanuts. So then they after the training they measure them as a group once and guess what? Their test scores go up. The group has improved in their knowledge and skill levels which you would expect. Problem is we don't know why. Could be because of the peanut farmers they were talking to down the road before they got the last
test. Or they could be figuring it out on the internet all by themselves. So that's problem number four. We just don't know if the training was responsible for the increases in knowledge and skill levels that they actually observed and measured. And then finally problem number five which is let's say they do it perfectly. They measure them before during and after the training. So you can see this nice trend of increases in skill levels and knowledge level. They also watch their incomes. They
look at their bank accounts and they notice oh this is great. Their incomes from selling peanuts has gone up due to the training before during and after. It's gradually gone up which is pretty good. But problem is there's no comparison group of people who also grow peanuts but did not attend the training.
So there could be another group of people in some other part of the country that are doing just fine teaching themselves how to grow peanuts and how to sell those peanuts by talking to other peanut farmers or whatever. So the problem there is we don't know if the project training that you paid for as a Canadian taxpayer was actually responsible for the increase in skill and knowledge levels of how to grow and how to sell peanuts.
And also the increase in incomes from selling those peanuts. We just don't know. It could be equal or it could be even lower even though we're seeing increases in incomes and knowledge and skill levels. It could be lower than the group that never attended the project training. So that's problem number five. So the podcast is going to focus in part one on a summary of those problems for the indicators for the performance measurement framework and where they fall.
Now there are so many outcome indicators in these projects. We will not have time to go through all of them but I will provide a summary of all of those indicators as to which ones fall into one of those five problem areas. So at the end of part one I will summarize the PMF or performance measurement framework and I will just cut to the
chase here. I will be concluding that the project in its current form with its current performance measurement framework is flawed and cannot support the international development organization making the claim that their project services are achieving their project outcomes. That's why we have this podcast to bring this out and provide solutions on how to improve
development evaluation. So at the very end of part one I'm also going to invite the international development organization to attend part two of the podcast where they will respond to this critique. But this is only a starting point for me as the host. I also will be inviting in part two on the podcast at least one and possibly two evaluation experts to also respond to my critique and provide alternatives to how to improve the evaluation of the project.
In part two it will take about 40 minutes so we won't be able to go through all of the outcome indicators but we hopefully will be able to select a few from each of the outcomes of the project. Usually these projects have five or six outcomes and then for each outcome they have two or three indicators. We'll probably just have to pick a few of them and go through them to make sure we cover all of the outcomes.
And at the end of part two we're going to answer the following question the host that's me and the evaluation experts and that is the question is the project in its current form with its current performance measurement framework which they are required to follow and develop before they get funding from the government of Canada is it adequate for the international development organization to make the claim
that the project is achieving its outcomes. My answer right now is no. There may be a few of those 12 organizations that you'll see in season one where maybe some of the indicators get it right but most of them fall into one of those five problem areas and for each episode I'm going to post on a website the project's performance measurement framework as well as a summary of how each of the outcome indicators in that performance measurement framework falls into one of those five problem areas.
So just to recap in part two of the episode I will be encouraging the international development organization to respond to my critique which includes offering alternative solutions to this problem of how to improve their evaluation of their project. The idea is to get a discussion going about how to improve development evaluation.
So you might be wondering what we plan to cover in season one like I mentioned we have 12 episodes introducing 12 international development organizations each one having a project along with its performance measurement framework and so these projects their cost to the Canadian taxpayer all 12 of them is around 177 million dollars. What sort of services are these projects delivering?
Well some of the projects we'll be covering focus a lot on training how to grow peanuts how to tutor midwives how to change men's attitudes to support women's rights and equality and raising awareness on accessing reproductive health services and gender-based violence services. These projects are being delivered in West Africa in particular Ghana, Ivory Coast, Senegal, Burkina Faso and in East Africa in Somalia, South Sudan, Tanzania and Kenya as well as in Asia, Bangladesh and Vietnam.
And finally you may be wondering okay you're going to listen to the podcast hopefully but in addition to that you may want to get a bit more involved as you can sense from the podcast there's a bit of advocacy involved where we're trying to advocate for improvements in development evaluation. The government of Canada spends over four billion dollars a year on development projects all around the world. You may want to send me an email regarding this podcast your comments and suggestions
on how the podcast can be improved. My email address is evaluate canadaaid at gmail.com and finally I'm always looking for evaluation experts who may wish to be on the evaluation panel in the podcast part two. And you can email me and I will be happy to send you the performance measurement frameworks for season one. We've got four already booked with evaluation experts but I can send you the remaining eight performance measurement frameworks.
And for season two we will be getting an additional five performance measurement frameworks. And if you're really interested and you want a particular project for us to look at on the podcast you can go to your search engine and enter global affairs canada project browser and up will come the platform where all of the projects thousands of them are listed.
And you can select a particular project that you have an interest in and you will notice that even though there's a project browser there's no performance measurement framework. It's not available and so ideally what we want to see is that performance measurement framework available on the project browser platform as well as the international development organizations putting their performance measurement framework for each of their projects on their websites.
Ideally both the blank ones that we're discussing on the podcast but as well the data that comes in on the outcome indicators that they've listed. The irony of course is as you may have figured this out is from this podcast most of those outcome indicators are flawed so it wouldn't be very useful to use the data coming in from those outcome indicators. As the saying goes garbage in garbage out. And that's the big issue that I am trying to raise
in this podcast. I look forward to hearing from you. Thank you for your time. Goodbye and stay tuned for episode one part one.