Welcome to the Improving Development Evaluation Podcast. I'm your host David Wand and welcome to season two. This is a trailer where we're gonna talk a little bit about some of the changes and difference and focus that we're gonna do in season two. And I first of all wanna apologize to the listeners out there where I had said two things that we were gonna cover 12 episodes, what I meant was 12 projects in season one when actually we only managed to cover eight projects. And I apologize for that.
And I also indicated that we would try to start season two in September, but there's been a few delays due to, what do you call it, economic circumstances. So I am now finally getting in gear for season two. So I thought I would give you some details about how we've changed it a bit for season two. And those changes are in just brief point form and then I'll go into more detail.
First of all, I just wanna reiterate for those who aren't familiar with the podcast and haven't been following it, the focus of this podcast is to expose those projects funded by the government of Canada, which is in the hundreds of millions of dollars, if not billions, that are funding organizations to deliver services in developing countries that are making claims of development outcomes. So we're not interested in looking at charity projects and there's a lot of charity projects out there.
For example, money being sent by the government of Canada through non-governmental organizations to, for example, pay school fees for children in Africa. That's a charity. They don't care after they pay for the school fees. Everybody goes home. We're not interested in looking at those projects. We're also not interested at looking at other projects that are also vital and very important and that is emergency funding. I like to call it emergency. People call it humanitarian.
It drives me crazy because when they say that, what do they mean? So this is money that's going overseas to keep people alive in emergency situations like in refugee camps for food, water, clothing, shelter, medical care, emergency medical care. That's fine too. That's really important, but that is not the focus of this podcast. The focus of this podcast is to expose those projects that like to wave the outcome flag and that is claiming that their project services are achieving these outcomes.
And as you've noticed from season one, there's all sorts of projects out there where they're not properly measuring and they're not properly evaluating these project services to support that claim. So that's the focus of the podcast. So what we're gonna do in season two that's a little different is we're going to repeatedly relate the flaws in these performance measurement frameworks that the government of Canada requires before they fund these projects for services delivered overseas.
We're gonna connect those flaws to the development assistance committee, DAC, DAC, evaluation criteria. And for those who are familiar with it, if you Google OECD, DAC criteria, you'll notice they come up with six criteria to properly evaluate a project. And those six are one, relevance, two, coherence, three, effectiveness, four, efficiency, five, impact, and six, sustainability.
But for my podcast, I've kind of done a brief review and we really are only looking at three of those six that keep coming up because in the trailer for season one, I identified five problem areas. And those five problem areas for these performance measurement frameworks fall into three areas under the DAC criteria. And they are one, relevance, two, coherence, and three, impact.
So we're gonna be looking at in detail how the performance measurement frameworks fall into those three problem areas related to the DAC criteria. So I'm not gonna go into the detail right now about that, but perhaps you'll get an idea of why it fall, they fall into those three areas. Particularly impact, because that's the big one. Because the impact definition is basically claiming that the project services and not other factors are indeed responsible for achieving the outcomes.
And that the project beneficiaries who show up to these services don't figure it out on their own through YouTube or talking to colleagues or whatever. They actually have learned the outcomes from the project, i.e. impact. The second thing I'd like to talk about that we're doing differently in season two is despite the fact that I'm Canadian, we can Canadians get access to information requests on US Agency for International Development Projects, USAID projects. So that's pretty cool.
So I'm gonna expand to the USAID projects and it took me a while, but a colleague down in the States told me through an evaluation group that the key magical word is activity MEL plan. So that's an activity monitoring evaluation learning plan. That's the document that is equivalent to a Canadian performance measurement framework. So what we're gonna try to do now is get some activity MEL plans for projects through an access to information request in the United States.
I already sent a couple of months ago an annual report from one organization that received $5 million from the United States government according to their annual report. So I sent it to USAID and I said, here's the annual report, 5 million bucks. Give me the activity MEL plan that or MEL plans plural that falls under this 5 million bucks. They took about two months, they got back to me and then they said, we can't find any activity MEL plan for this 5 million bucks.
So I then sent back and I said, well, if you can't find it, can you tell me where this money is coming from? And that was about a month ago. And they said, by the end of October, they said they'd get back to me on first of all, finding where the money came from and hopefully secondly, giving me the corresponding activity MEL plans so that I can go through them and do the same thing that we did in season one and see if any of the outcome indicators don't smell right.
So that's another thing we're gonna be doing. And number three, the format changes. Instead of just doing two parts, where part one I described the project and then in part two, we go through all the outcome indicators with evaluation experts, two different changes there. I'm gonna combine parts one and part two, which hopefully will lead to me covering more projects in season two. And also I've dropped evaluation experts because it's basically the same issue.
So that's what I'm gonna do there, just do it by myself.
However, since I have an ambush mentality, if you haven't noticed on this podcast, where we're trying to confront organizations that are waving this outcome flag when they should be burning it because they're really aren't measuring the outcomes and they're either doing this fraudulently and knowingly or they like to claim, oh, we don't have the money to do it properly or we don't have the technical expertise, which is another excuse they make.
And I do wanna point out that this is not some right wing conservative podcast in the sense that we wanna cancel aid. Yes, people on the right side of the spectrum wanna reduce aid relative to the left side of the spectrum, but the left side of the spectrum should also be concerned because if they want aid to continue and receive continued political support from people like me, cause I'm on the left side, doesn't matter.
We should all be concerned that our tax dollars are making a difference, i.e. impact. So it has to be properly measured and evaluated. It's not some right wing concept saying that you want to have your programs that you pay for properly measured, properly monitored, properly evaluated. Everybody should do that because in the end, the irony is that if you don't show that that the programs are being effective and having impact, the conservatives will use that as an excuse to scrap the program.
So it's best that if you believe in it, that you support it by showing the evidence that the programs actually work. So that's important also. And finally, a few more points.
What we'll do in relation to that is that I will continue to, if I find some organizations that have multiple projects and they're doing the same wrong things where they're not properly measuring the outcomes using proper outcome indicators and proper evaluation design, then if I find that and I find it repeated going on through various projects for the same organization, then I may set up an ambush podcast where I invite them to respond to my critique
and I also will have evaluation experts on board that agree with me. So let's be honest about this. We are confronting these organizations who are making these claims when these claims are not being supported by the data that's coming in, let alone the performance measurement framework or the activity MEL plan, USAID that they're using.
So we'll still do that, but I'll be more selective in the organizations I select to invite to attend a part two of our podcast where they can respond to the critique that we do. I will, just like season one, I will continue to offer to anyone if they want to email me at evaluatecanadaaidatgmail.com if they want any of the performance measurement frameworks from season one, as well as my Excel summary of the outcome indicators and how each one of the outcome indicators are flawed.
I'm happy to send those out. And I did receive a few emails and I'm happy to continue to do that.
And I will also continue since there's clearly an activist advocacy intent from this podcast, which is to get these hundreds of millions of dollars of projects paid for by the Canadian taxpayer to start properly evaluating them, is I will continue to send my critique and the episodes to the Minister for International Development, as well as the shadow critics for the Conservative Party, the New Democratic Party and the Green Party, because like I said earlier,
proper evaluation should not be an issue of the right or the left. It should be an issue for all Canadians to be concerned, to make sure that their dollars overseas are making a difference.
And finally, I wanted to invite another format change is that there is a growing industry in the evaluation industry, especially in international development for three types of experts that are being used to evaluate projects, which I'd like to learn more about because I'm still having issues with how these three different evaluation techniques actually properly, validly measure the achievement of outcomes, and in particular show that the project has impact.
That is, in the absence of control groups or comparison groups or repeated measures, how does the project show that it is actually achieving impact, making a difference, rather than other factors? And those three areas of expertise are outcome harvesting, most significant change, and appreciative inquiry.
So those are three evaluation techniques that have come up quite frequently on various platforms, claiming that they are proper ways of evaluating impact of a project, or at least showing relevance or coherence, which we'll talk about in more detail when we get to reviewing these performance measurement frameworks or the USAID activity MEL plans.
So if there's anyone out there who has expertise and would be willing to come on my podcast, I'd be happy to interview you so you could explain to our listeners how outcome harvesting, most significant change, or appreciative inquiry actually evaluate these projects properly. So feel free to email me at evaluatecanadaaid.gmail.com and we'll set up a time, because I would love to put that as a separate episode on the podcast. So thank you for your time and thank you for listening.
And soon we will begin introducing new projects from the government of Canada and hopefully from USAID. Bye for now.