I'm your host David Wand and in this episode 2 part 2 we're going to be talking about the Canadian Bureau for International Education, their project, cost of $5 million to the Canadian taxpayer in Canada.
And just to give you a recap of part 1, I reviewed the 7 outcomes and the 11 outcome indicators for that project and came to the conclusion that the Canadian Bureau for International Education could not make the claim that they are properly measuring the outcomes using the outcome indicators in its performance measurement framework.
I also contacted the Canadian Bureau for International Education inviting them to attend this podcast part 2 to respond to my critique of their performance measurement framework. However, they declined to attend this podcast and they also declined to send a written response to my critique.
But fortunately today we also have with us an evaluation expert from Amman Jordan and her name is Hyatt Asgar and she holds a graduate diploma in public policy program evaluation from Carleton University in Canada. Welcome Hyatt, how are you? Thank you, hi David, glad to be with you today. Good, and we're also glad to hear from you.
And what we're going to do now is go through each of the outcome statements one by one and then we will have a discussion about why their outcome indicators are not properly measuring those outcomes. So let's start with outcome number 1, increased leadership skills of female and male African leaders of tomorrow, scholars in public administration and public policy. Hyatt, what outcome indicator do you want to choose and talk about?
Yeah, I'll speak about the indicator number 2, the percentage of schoolers reporting that leadership capacity building events have increased their ability to take on new leadership responsibilities in public administration and public policy. Tell us what's wrong with the indicator in measuring the outcome. So I'll share some observations from my personal opinion and I think what caught my attention for these indicators is that we are asking schoolers to report.
So we're speaking about self-reporting, which in like you could see it in other monitoring and frameworks, it is used, but is that enough to have this self-reporting as a source of information alone? This is what is called the reporting bias. So you are asked the students to report on their ability, how they can see their leadership skills.
You expect that people will be judging themselves, whether telling you the truth from their own perspective, whether underestimating themselves or overestimating themselves. So the self-reporting, I cannot say that it is wrong, it can be used, but it should be supported by other sources, other assessment methods, like for example, having a different measure to look at their leadership skills using some sort of leadership assessments.
And here they are speaking about so many available leadership assessments worldwide that the organization may choose to select. Also, if they are focusing on a specific sub-factors or sub-leadership factors, they can build their own leadership assessment. But what is good about having a supporting measure that they can use a scientific approach and then they are also welcome, they could use the reporting as a second source and they compare the two together. So this is my first observation.
Yeah, that's very good. Thank you. And just to recap, of the 11 outcome indicators in this Canadian Bureau for International Education Performance Measurement Framework, 11, 8 of them were having this problem of self-reporting bias and 2 others were having this problem of no comparison groups. And we'll get to those later too. So the second outcome is an interesting one.
It's increased access of female and male African leaders of tomorrow scholars to opportunities for exchange and collaboration within the framework of Canadian Pan-African Public Administration public policy networks. Now what's interesting about this one is it's really not an outcome. They want to increase access for these scholars for opportunities to exchange.
And that is basically what the project is doing by sending them from their African countries over to Canada to complete this master's program at public administration. So I think they have increased their access, but that's not an outcome. That's part of the services they're getting by being in the project. So we're not even going to look at the indicators for outcome 2.
Now outcome 3, however, is an outcome because they expect these African students studying this master's program at public administration to have, quote, enhanced understanding of female and male African leaders of tomorrow scholars respecting public administration, public policy development, and implementation processes, including the integration of gender equality considerations.
Now this one's interesting, as I mentioned in part one, because there's the self-reporting bias obviously, because they're asked, and the indicator is. Percentage of female and male African leaders of tomorrow scholars reporting that their mentoring relationships with Canadian leaders, practitioners, have strengthened their knowledge of public administration, public policy development, and implementation processes, including the integration of gender equality considerations.
So again, we have the same problem, but they're trying to measure strength and knowledge of not just public administration, but specifically to gender equality. And they're doing this by self-reporting. And again, that's not a good measure of whether or not they have, quote, actually enhanced their understanding. They could have enhanced their understanding, but we have this problem with the self-reporting bias.
And I had mentioned in part one that there are lots of gender equality indices out there where the United Nations Development Program, UNDP, is measuring these at the country level. What would be of interest to me is if they measured their ability to calculate this index for the country where they come from in Africa to see how they're doing on this gender equality measure.
And that would have been a better indicator for me in terms of measuring their enhanced understanding of, quote, gender equality. Can I add one small point here? I'd like to support what you were saying. What is interesting about using available gender equality indices is that, especially here where I was speaking about Africa and Canada, so different countries, it provides them a room also to benchmark, to compare with other countries.
So it's not only looking at your own schoolers, but these indices are global. They are used in multi-other organizations. So it would have given them the opportunity to compare even with their learning and their outcomes with other organizations and other countries as well. Right. Yeah. No, that's good. And that, given that this project, I think, is attracting African students from, I think it was 23 African countries, as I described in part one.
Yeah, that would be interesting, especially if those students as part of their MPA program, because we don't know in their courses in Canada if they had to take, if they had to learn how to calculate this indices, I had to learn it when I was doing international development. But it would be interesting as an exercise for the students who are in this project to do this comparison.
And I used to do it when I was teaching international development at the University of Toronto, where each student would pick a developing country and we would go through a variety of indices such as the gender equality measure and we would graph it to see the comparisons between developing countries. Outcome number four is increased knowledge of female and male African leaders of tomorrow, scholars in governance, public administration, public policy and gender equality.
And I've already sort of alluded to this in the previous outcome. The indicator here I've chosen is the percentage of female and male African leaders of tomorrow scholars who have successfully completed a master's level public administration, public policy program in Canada. Now you would think logically that completing this master's program in public administration would be adequate to claim that they actually indeed increase their knowledge of public administration. Nothing wrong with that.
But one of the issues here is comparison groups. There could be other master's programs in public administration where African students have attended it in Africa because there's a lot of African universities that offer the same MPA program. And it would be interesting to compare levels of knowledge of public administration or public policy with a comparison group and even specific to their ability to calculate these gender equality indices.
So the first one indicator, even though graduating is a good indicator of increased knowledge, I would think you would have to go a step further and say there may be MPA programs out there where students are also increasing their knowledge and at even maybe a greater rate than the students that are in this project. So we need to ideally have a comparison group where we can show that this project actually had a better level of knowledge acquired by the students.
The next outcome, number five, is increased effectiveness of female and male African leaders of tomorrow scholars to provide leadership within public administration, public policy networks, Canadian and Pan-African networks. And the indicator that I'll be looking at is percentage of female and male schoolers who report making a meaningful contribution to public administration, public policy development in key sectors in sub-Saharan Africa.
So the first thing before looking at the indicator that I think I would like to comment on as the outcome itself, because measuring effectiveness and planning to increase effectiveness is in itself is a very big outcome, is broad, usually, and for evaluators to measure effectiveness of a training or of a program, we're speaking about multi-month evaluation and even might reach one-year evaluation. So it's measure that for an outcome indicator in my perspective might not be enough.
And again, we have the self-reporting bias here again. So it shows as David mentioned, this is another indicator where we have reporting bias. Oh, sorry to interrupt, but is it...we've sort of skipped here. This is outcome six, increased effectiveness of female and male African leaders of tomorrow scholars to contribute to public administration, public policy development in key sectors in sub-Saharan Africa.
And the indicator is percentage of female and male scholars making a meaningful contribution from the performance measurement framework. Do they have to report this themselves? Is that what you're saying? The wording of the indicator and the document they have is percentage of female and male scholars who report making a meaningful contribution. Excellent. Okay. So thank you for that. So yeah, who report? So there's a self-reporting bias. Okay, great.
And increased effectiveness is not going to measure that. Yeah, and another important thing is the use of the word meaningful contribution. So how could we know that they are doing meaningful contribution? What does meaningful contribution mean to them? Does it mean like...I think I haven't found any definition of that. Maybe it's in their documentation, but at least in the BMF it's not there.
I would expect like that there will be a clearly that such meaningful contribution should be clearly defined and then they can measure it whether through what applies to the African context. Like they can say, okay, the meaningful contribution could be to what extent they are doing quality written products or briefing notes or white papers in their work, or maybe to what extent they are pushing the agenda, the policy agenda in a specific sector.
So I know it varies based on the context, but at least there should be boundaries or a clear definition of it. So that they can, even if we're speaking about self-reporting, so that student can't report in it. But again, preferably it can be looking at actual tangible deliverables that these students are doing. Right.
Yeah. They should have a definition of what they mean by meaningful contribution, maybe a certain number of briefing notes or white papers or whatever analysis that they're doing when they graduate from the MPA program. That would be a good measure of meaningful contribution, at least try to attempt to measure it a bit more objectively. Right.
Now, outcome number five, just to go back, increased effectiveness of female and male African leaders of tomorrow scholars to provide leadership within public administration, public policy networks, Canadian, pan-African networks. And we have two indicators there. I think you had mentioned you would want to talk to the number of inputs or the percentage of male and female scholars actively involved in leadership functions within public administration, public policy networks.
My understanding for that one is self-reporting also, right? Yes. And sorry here, it seems that there are some confusion in terms of the order, but I can speak about the number of inputs such as paper studies, presentation lectures provided by female and male schoolers to public administration networks. So here it is, we are counting number of papers and to measure such a high level outcome, would the number of papers in itself be enough?
So I could develop as a school or even a public policy official five blogs while another person might come up with a more meaningful policy paper that is being provided to the government. So number in itself, like I prefer that if they have it to look at more a different way of put this indicator and also the comparison group, like how can they, is that, are there any other group of people or graduates? How is that about like the papers they are developing? Are they of different quality?
Are the graduates of this program providing more quality papers, more quality or more like papers that makes real change to their entities they are working on? So yeah, I don't know if you have anything else to add to this also, David. The only thing I would add is they talk about number of inputs and we talked about it earlier.
This is kind of in the right direction where they are suggesting if these students when they graduate can produce high quality knowledge products, what you could do is maybe set a standard of, for example, in Canada there is the institution of public administration of Canada and I think they still have an academic journal. There is also a magazine called Policy Options.
There could be equivalent sort of magazines or academic journals in the African country where the students come from where they could track their contribution to these particular journals as a measure of increased effectiveness. But to be fair to the Canadian Bureau for International Education, it's a bit difficult, but they should try at least to have an attempt to make a better indicator rather than just saying number of inputs.
They could keep it at number of inputs as long as they have specified criteria or what's considered to be a high quality input and they could apply that across all 23 African countries where these students come from. And then they could argue that when they started the program they didn't produce any of these inputs and then when they finished the program and further on they produced these inputs and that could be a measure of increased effectiveness.
Yeah, and the last outcome is outcome number seven and it is strengthened economic and social development, key public administration and policy sectors in sub-Saharan Africa. And the measure here they have only one indicator. It's the percentage of female and male African leaders of tomorrow scholars who hold senior level decision making management positions in sub-Saharan Africa.
This is fine as an indicator, but the challenge here again is graduates are asked if you look at the performance measurement framework, this is why you need to get the performance measurement framework to actually see the details of how do they come up with this indicator. They ask the graduates themselves to report where they are, the entry, middle or senior level in the government.
And they shouldn't be asked to report especially when the criteria for what's entry level, middle or senior level is not defined. And they should try to do that across all 23 African countries. And it's possible to be able to do that. And then what you do is interview the graduates and you ask them through a checklist of a series of questions and then from there you could put them in the box whether they would be considered to be entry, middle or senior level.
And the interesting thing I would add to this is that's the first thing they have to do to improve the indicator is get an objective measure and then follow up with the graduates when they're back in their African countries in the civil service and ask them, you know, a series of questions. I'm repeating myself to get into these three boxes, entry, middle or senior level.
But the other thing they should be doing is comparing these graduates of MPA programs with in the same country for each of the 23 African countries where these graduates went back, they should also be asking graduates from MPA programs outside of this project and see how well they're doing. Same level of education.
They just find your MPA students who graduated working in the same government in the same African country and ask them also the same questions and to see what percentage of those graduates are occupying senior, middle or entry level positions because again, without a comparison group, this project may be not doing as well as graduates from MPA programs outside of this project, which is fine because what you're doing here is providing these
Africans with an opportunity they otherwise may not be able to get because the tuition at MPA programs is very expensive for foreign students. So the issue is if they're going to make the claim that they're strengthened development by increasing the percentage who hold senior positions from this project, they need to use a comparison group.
And I know from personal experience working in Rwanda and Nigeria at the National Institute of Statistics, I worked with very bright, talented senior managers who were graduates of master's programs in statistics from local universities as well as international universities also. So there's a huge variety of civil servants in the National Institute of Statistics in Rwanda as well as in Nigeria who had education all around the world and including in Nigeria.
So, I think they need to show that this project is even better at strengthening economic and social development of their graduates. But another option is to just say, look, we're in the business of charity and we're just going to make sure that these students get their tuition paid for by MasterCard Foundation, which is what MasterCard Foundation is doing here.
So you could argue that maybe this project isn't necessary at all, the $5 million, they just need the $5 million or whatever it is for the tuition to pay for the 120 to 130 students. So that'll be an ongoing debate. But that's the idea.
Yeah. I have one, like I'm curious to know from this outcome, like on this indicators, how they are seeing the link between this indicator and the outcome because measuring the strength in economic and social development, how we envision this to happen by having these seniors in these positions.
So I think there might be something in between needed and especially which could be another also question that I'm curious to, like I cannot find an answer to as they are addressing 123 to I think 130 students from 23 countries. So if we do the formula where maybe in some countries they will only have five graduates, so do we expect these five will have that contribution to strengthen the economic and social developments? Will they have the ability to do so?
So some like more clarification, further information might be needed and other indicators might need to support that as well. Very good point. It ties in nicely with a comment I made in part one, which is that Canadian immigration policy right now is to encourage a lot of young, bright, talented students from developing countries to come to Canada and complete their graduate or undergraduate education, graduate and stay in Canada.
And that conflicts directly with the objectives of this program, which is to strengthen the development of key public administration and policy sectors in sub-Saharan Africa. So this is another interesting point, which is that a lot of graduates in international development in Canada are perfect for staying in the country because they have the skill set, the high levels of education, they're young, and that's part of the Canadian immigration policy right now.
And it would conflict exactly with what you've said, where maybe they only have five in a particular country and even worse, some of the students graduate in Canada with their MPAs and they don't go back because there's an aging population in Canada where there's a huge demographic boom around the 60 to 65 age group that are retiring from the public sector and the age group behind them is smaller because of the birth rate. So that's another interesting point to make.
And it raises the whole issue again of is this really a development project? And it could be just a charity project and there's nothing wrong with having a charity project by giving these students an opportunity to avoid having to pay $18,000 Canadian in tuition, foreign international student tuition for an MPA program. I think if you go on the websites here in Canada, I think that's what the foreign tuition at your university, Carleton, I think it's about $9,000 a semester for foreign students.
So it's very expensive. So you could argue, you know, that's fine, but it conflicts with the outcomes that they're claiming in this project. Yeah, yeah. Now that we've done all seven outcomes, we took out outcome number two, because it was access and it's actually an output. Are there any other outcome indicators that you wanted to talk about or are we done? Okay. Yes, there's one under outcome number one. Case study experience. That's right.
Yeah. Yeah. Percentage of female and male African leaders at Tomorrow Scholars reporting that the case study experience has increased their capacity to address policy changes within their home environment. And that indicator was for measuring increased leadership skills. Yes. So what I want to mention first is that what I liked, like this initiative looks at the different learning development stages.
So it looks into the direct attendance of events, looking into mentoring, applying the learning through case studies and then providing mentoring along the way. These indicators in particular, like there is this case study. I did not get from the BFM how exactly they're doing that, even with the outputs that it seems that they will be applying, doing, exploring or doing a case study, applying the case study experience on the country.
And there are some audience who will be, I think will be looking into their presentation, but it's not 100% clear. But if they are applying that into a case study and to their countries, it will be good to know if are the people like, are there external people who are evaluating them? Because this indicator again is a self-reporting and to self-report, like to report on their own experience with a case study will give us some information from them on the learning along the way.
But was that really adding value to their leadership skills and how you need someone else to provide you this assessment and someone with experience and with external backgrounds? And yeah, so this is generally on this indicator. So, yes. And it's also saying within their home environment. So I'm wondering how they do that. Yeah, they would have to be back in Africa, right? Exactly. This is why I said it's not 100% clear.
So I think there are some like, there needs to be some more clarity in the PFM in general. And this is a general observation. No that's true. Yeah. So I think we've covered the performance measurement framework and thank you for your input, Hyatt. And what we're going to do now is we will send this podcast part two, episode two, to the Canadian Bureau for International Education. But more importantly, as noted in the trailer for this podcast, we need to advocate for two things.
We'll be sending this podcast recording to the Minister for International Development, the Honorable Ahmed Hossein, as well as the shadow critics from the Conservative Party of the New Democratic Party on international development. Because this is a universal problem. It doesn't matter what political party you support.
We need to have proper evaluation of development projects funded by the Canadian government, regardless of who the Canadian government is, Conservative, Liberal, NDP, Green, whatever.
So we'll be recommending one, that performance measurement frameworks be required to be put on the websites of the organizations such as the Canadian Bureau of International Education, that they put their performance measurement framework for all their projects funded by the government of Canada on their websites, both the blank ones and the data that's going to be coming in on these indicators.
And number two, we want to recommend, since the podcast has been raising this, that the government of Canada follow their own guidelines on how to properly design outcome indicators to properly measure the achievement of the outcomes for the projects that they fund. Because we're talking about five billion, between four and five billion dollars a year, the government of Canada funds international development projects all around the world. So we'll be advocating for that.
Hopefully someday, everyone will be able to see a performance measurement framework on every organization's website. So Hyatt, thank you so much for taking your time to provide your expertise and input and perhaps you'll return again for another episode because I've got another 12, well I've got a total of 12 lined up for season one and then I have another 12 lined up for season two. So we'll be doing this for a long, long time.
Yeah, no, sure, and thank you for your efforts and I think you are doing something very important. And as mentioned, there are a lot of good things in this PMF, but also again, there is room for improvement. There are ways to present even the way that this PMF is presented if it's supported by a more easy to read theory of change or visual, will even maybe answer some of the questions that the reader might have. But yeah, I really appreciate your time and involving me in this.
So thank you so much. Well, thank you and we'll keep in touch and feel free to also, I want to add that if anybody listening out there and I keep repeating this, I'll just say it one more time.
If you're interested in getting any of the performance measurement frameworks that we've discussed on any of the episodes along with the summary of the outcome indicators and how they fall into one of these five problem areas of not properly measuring the outcomes for the project, you can send me an email at evaluatecanadaaid at gmail.com and I'd be happy to send them to you.
And I'm always looking for bright, talented evaluation experts like Hyatt who would be interested in being on future episodes of the podcast. I can also send you the future performance measurement frameworks that I have that I'll be featuring in future episodes. So thank you again and stay tuned for episode four. We've already done episode three, but episode four will be coming up also.