Welcome to the Improving Development Evaluation Podcast. I'm your host, David Wand, and welcome to Season One, Episode Two, Part One. In this episode, we're going to introduce you to an international development organization. That international development organization is the the Canadian Bureau of International Education. You can learn more about the Canadian Bureau of International Education if you go to their website at www.cbie.ca. Their headquarters is located in Ottawa, Canada.
The project we're going to be talking about today in Part One is for Episode Two. The Canadian Bureau of International Education has been funded by the government of Canada to deliver this project, and it is called African Leaders of Tomorrow Scholarships Fund. This project is being delivered in Canada, and the cost of this project to the Canadian taxpayer is $5,127,124. Of course, it could be a little less, I think. It depends on the actual expenditures.
And in this Part One, I'm going to give you a brief description of the project services that are being delivered and a critique of the Canadian Bureau of International Education's PMF, Performance Measurement Framework, that they have developed to measure the project's performance. That is, the achievement of the project's outcomes using the outcome indicators in that performance measurement framework.
But before I describe the project services that are being delivered for this project by the Canadian Bureau of International Education in Canada, I'm going to go to my summary and conclude that in this performance measurement framework for this project, there are 11 outcome indicators that are measuring six outcomes. There is a seventh outcome, but its outcome indicators will not be reviewed, as I will explain a bit later, because this outcome is actually an output or service.
It's not what you expect after you deliver the service. It actually is the service. So that's why we're not going to look at it. So let's start. My review of all 11 outcome indicators has led me to conclude that all of these outcome indicators are not measuring the six outcomes.
So the organization, the Canadian Bureau of International Education, cannot make the claim that it's achieving its project outcomes with the services they're delivering in the project simply because their outcome indicators are not properly measuring the outcomes for this project. As you probably know from the trailer for this podcast, those 11 outcome indicators fall into one of five problem areas.
Now for this project, eight of those outcome indicators fall into the problem area of how do you feel self-reporting, where people are asked how they feel about achieving the outcome rather than being objectively measured on whether they achieve that outcome.
Another two of the outcome indicators fall into problem area number five, which is there's no comparison group of people outside of the project that are also measured on the same outcome, which means even if people in the project are achieving the outcomes, we don't know if they're achieving them any better than another group of people that could also achieve the outcome outside of the project. Some people refer to this with a fancy word called impact.
But I won't bore you with that word. I'll just stick to the facts. And finally, we have one outcome indicator that is falling into problem number three, where the outcome indicator is measuring something that is not related to the achievement of the outcome at all. So now that I concluded that the PMF is flawed, that means the performance measurement framework is flawed. Let's get into more detail about the project services.
And just to remind you, if you wish to get more detail about this project, you can go to your search engine and enter Global Affairs Canada project browser into your search engine. And what will come up is a platform with all the projects listed there, you can go to the partner list and select Canadian Bureau of International Education. There you will see a description of this project.
If you wish to get the Canadian Bureau of International Education performance measurement framework for this project, as well as my summary of the 11 outcome indicators for this project and where they fall into one of those five problem areas, you can email me
at evaluatecanadaaid at gmail.com, and I will send it to you. Also, in that summary, I will include the link to the project description at the Global Affairs Canada project browser platform where you can find a description of the project and the services they deliver. And as you know from the trailer, you will not see on that platform at Global Affairs Canada, the performance measurement framework. Hence, I think the need for this podcast. So let's start with a brief description of the project.
The target group that is receiving the services delivered from this project are Africans who are entering a master's program in public administration in Canada. Between 120 and 130 students are selected and placed into this master's program in public administration in Canada with these students coming from 23 African countries.
The project delivers the following services to these students. Number one, mentoring by Canadian public administration practitioners to those students from Africa, who are attending a master's program in public administration in Canada. Number two, delivering learning events to these students.
Number three, providing online platforms for the students so that they can network with other students and other people. Number four, matching these students with Canadian practitioners for internship placements. Number five, developing guidelines to support these students when they are interns.
Number six, monitoring these students in their academic programs. And finally, number seven, a variety of administrative services that the project delivers, such as helping the students get their study permits, paying for promoting the program to get international student applications, using a website, processing their applications online, and selecting and placing the candidates in these master's programs in public administration in Canada.
Additional funds are provided by MasterCard Foundation to pay for the foreign international student tuition fees for this project, which are quite high, as you probably know if you've been following the news, like around $18,000 a pop per year. So, there you go. So that gives you an idea of the services the project delivers. Now, let's get into the outcomes.
Let's get into the outcomes for this project. There are seven outcomes, and they are outcome number one, increased leadership skills of female and male African leaders of tomorrow scholars and public administration and public policy outcome number two, increased access of female African leaders of tomorrow scholars to opportunities for exchange and collaboration within the framework of Canadian and Pan African public administration public policy networks.
This is not an outcome, but direct services, increased access with no expected outcome from accessing those services. Therefore, its indicators will not be reviewed and this outcome will not be reviewed. So we have six outcomes that will be reviewed.
Outcome number three, enhanced understanding of female and male African leaders of tomorrow scholars, respecting public administration, public policy development and implementation processes, including the integration of gender equality considerations. Outcome number four, increased knowledge of female and male African leaders of tomorrow scholars in governance, public administration, public policy and gender equality.
Outcome number five, increased effectiveness of female and male African leaders of tomorrow scholars to provide leadership within public administration, public policy networks, both Canadian and Pan African networks. Outcome number six, increased effectiveness of female and male African leaders of tomorrow scholars to contribute to public administration, public policy development in key sectors in sub Saharan Africa.
Keep in mind that one again in sub Saharan Africa, so they're expected after they graduate to go back to their home country and work in the public sector, the government sector in their respective country. Outcome number seven, strengthened economic and social development of key public administration and policy sectors in sub Saharan Africa.
So, part two of the podcast will cover at least six outcome indicators to ensure that all six outcomes, not including outcome two, because it's increased access that's not an outcome, that's an output. And it is not an outcome of the project are covered. So, a Canadian Bureau of International Education representative will be invited to respond to the evaluation panels critique of those outcome indicators and provide alternative methods of evaluation for the project.
At the end of part two, the evaluation panel will be concluding as to whether the Canadian Bureau of International Education can make the claim that its project services are achieving its project outcomes. As host, I conclude that the Canadian Bureau of International Education cannot make this claim. But I want our listeners to hear what the Canadian Bureau of International Education has to say, as well as the other evaluation experts.
Now I'm going to look at just three outcome indicators to give you an idea of why they fall into one of these five problem areas and in part two, we are going to make sure we go through all six outcomes, looking at least at one outcome indicator for each of those outcomes.
So, let's look at the outcome number three enhanced understanding of female and male African leaders of tomorrow scholars, respecting public administration, public policy development and implementation processes including the integration of gender equality considerations. One of the outcome indicators to measure this enhanced understanding is percentage of female and male African leaders of tomorrow scholars, reporting that they're mentoring relationships with Canadian leaders.
Practitioners have strengthened their knowledge of public administration, public policy development and implementation processes, including the integration of gender equality considerations. Measuring strength and knowledge can be best done objectively through testing, rather than self reporting, especially with something as specific as quote, gender equality, which needs to be clearly defined.
So, self reporting based on a mentoring relationship would not be a valid way of measuring that knowledge. It would not be a valid way of measuring a strengthening of that knowledge. Let's look at the outcome number four increased knowledge of female and male African leaders of tomorrow scholars in governance public administration, public policy and gender equality.
One of the outcome indicators for this outcome is percentage of female and male African leaders of tomorrow scholars who have successfully completed a master's level public administration, public policy program in Canada. Now on the surface you would think this indicator would be a good measure of increased knowledge, since a master's degree in public administration. After all, is directly relevant to knowledge specific to quote governance public administration and public policy.
However, two points to keep in mind here are that gender equality is very specific. So we need to look further to see of courses specific to gender equality were completed and fit with gender equality policies and programs back in the students African country, where they are expected to return to after graduation to use their
master's degree in public administration to contribute to and I quote governance public administration, public policy, getting the NPA would not be considered directly related to quote increased knowledge of gender equality, unless the NPA required the completion of courses specific to gender equality. For example, with these students be able to calculate the United Nations Development Program UNDP gender inequality index for their country, using a particular data set.
And would they be able to use that calculation to support their development of policies to address gender inequality. This would seem to be a good measure of their quote increased knowledge of gender equality. Finally, a second point is current Canadian immigration policy encourages international students to remain in Canada, after graduation.
This is a good example of how these immigration targets, as these immigrants are young, highly educated with Canadian master's degrees in public administration, and have become accustomed to Canada and offer a skill set to replace the large group of public servants in Canada who
are in the process of being able to apply for a degree in public administration. This conflicts directly with outcome number six of this project, which wants these masters graduates in public administration to return back to their African country, who and I quote contribute to public administration in key sectors in sub Saharan Africa.
This may raise the need to introduce the indicator here in the project percent of NPA graduates who return to their home country, and are employed in the government sector in their home country.
This will become seven strengthen economic and social development of key public administration and policy sectors in sub Saharan Africa graduates themselves are asked if you look at the performance measurement framework for this indicator to self report their quote level of position in Africa, as either entry, middle or senior position in Africa.
These positions should be defined and applied across all 23 African countries, where these students are coming from, and then graduates could be interviewed based on those criteria, rather than self reporting, that would be a better way of seeing if they are actually in the senior levels of government. Also, what about masters graduates in public administration outside of the project.
There could be thousands of Africans who go to universities in their own countries graduate with NPAs. They could be in higher senior positions than this project, showing better strength and development of the public sector than this Canadian Bureau of International Education program. This outcome indicator has self reporting bias.
This is a very good example of how this program is being used in the public sector compared to other NPA graduates that did not participate in this project. So those are just three examples from three outcome indicators that fall into those problem areas. If you have any critical feedback, any critical criticism, or critical criticism or critical critique an episode, feel free to email me at evaluatecandidateatgmail.com.
delivered by the Canadian Bureau of International Education, as well as my summary of all of the outcome indicators on which problem area that each of the outcome indicators falls into, feel free to email me.
And finally, if there are any evaluation experts out there who'd like to be on the podcast to look at other performance measurement frameworks, and in particular the outcome indicators, to see if they fall into these five problem areas, email me and I'd be happy to send you the remaining performance measurement frameworks that we have lined up for season one. We have 12, where we don't have evaluation experts so far.
And what we're going to do now, at the end of this part one, is send it to the Canadian Bureau of International Education representative, and invite them to attend part two of this episode, to respond to this critique here in part one, along with an evaluation expert that will also attend part two of this podcast, to also provide a critique of their performance measurement framework.
Thank you for listening, and stay tuned for Canadian Bureau of International Education, part two, episode two, season one.