So, welcome to the Improving Development Evaluation Podcast. I'm your host, David Wand, and in this episode, we're going to feature Humber Polytechnic International Development Institute. You can learn more about the Humber Polytechnic International Development Institute if you go to their website, which will be in the episode notes. It's humber .ca forward slash global forward slash international hyphen development hyphen institute hyphen IDI.
And we're going to look at a project that they received $4 ,950 ,000 from the government of Canada. That's you guys, some of you listening, the Canadian taxpayers. And it's for a project that they delivered in Bhutan. called Education and Skills Training. And I'll also put in the episode notes, the link to the project description. Of course, I'm going to give you a bit of background on what this project and the $4 ,950 ,000 services
it actually delivered. And as usual, when it's done, I'll be sending my critique, this link to the episode, along with the performance measurement framework. to the Humber Polytechnic to invite them to respond to my critique if they wish. And of course, I will always send it to the minister responsible for international development, which is I think called the Secretary of State for
International Development. I'll also send it to, of course, all the other political party shadow critics responsible for international development. And I'm always recommending... that instead of us trying to chase the performance measurement framework, that all of these agencies, including Humber, Polytechnic, put on their website their performance measurement framework, as well as the data, which we aren't getting right now.
But I do have some good news. There is an international organization that actually is starting to do it. So in future episodes, I will be critiquing their performance measurement framework that is on their website. So without further delay, let's get into this project in Bhutan. And one last point. If you're listening to this podcast on Spotify, Spotify has now added a comment section. So feel free to leave your comments in response to any of my episodes that are up there. I look
forward to reading your comments. And if you ever want any of my performance measurement frameworks along with my critiques, that are produced in writing. You can send me an email at evaluatecanadaaid at gmail .com. That'll be in the episode notes. And feel free to request, and I'm happy to send them to you. So this project in Bhutan, delivered by Humber Polytechnic International Development Institute. They basically have six target groups that they deliver services to with this $4 ,950
,000. Target group number one are technical vocation institution teachers. Target group number two, technical vocation institute curriculum developers. Number three, technical vocation industry representatives. Number four, technical vocation institutes. Number five, all the staff in the technical vocation institution. And number six, parents and community leaders at large. So you might be wondering,
what services do these target groups get? Well, the technical vocation teachers, they're receiving training on gender equality to incorporate within industry. They're also receiving training on career counseling for students, and in particular women students and students who have disabilities. They're also being trained on ability to coach students in self -employment to be gender sensitive and environmentally responsible. The industry
staff target group. Industry representatives, they're being trained on how to best accommodate women and people with disability students when they graduate from these technical vocation institutes. A third target group, technical vocation institute curriculum developers, they're being trained by Humber Polytechnic to be able to incorporate gender equality into the student curriculum.
And the technical vocation institutions by themselves as another target group, they are, it's not really training, but it just says in the performance measurement framework that their job is to set up incubation centers to enable students to pilot business ideas. They're also responsible for, with this funding, to set up resource centers. to deliver knowledge to students. Another target group are all of the staff at the Technical Vocation Institutes where they do this training in technical
vocations. That training is on gender equality, also on gender sensitive, disability inclusive admissions, advising and counseling, and also gender inclusive recruitment. And finally, the last target group which are parents and community leaders in the public at large, is promoting technical vocation to them for people with disabilities through events. And also for women, promoting technical vocation to women through education
fairs and education forums. So that gives you an idea of who is getting these various services and these various target groups. So before we get into... the outcomes and the outcome indicators to see if all these wonderful services were achieving the outcomes, we need to digress a bit and say to maybe the staff, although Bhutan is not really that hot and humid in the global south, perhaps. I've been to Nepal, but I haven't been to Bhutan. There may be times. where it is hot and humid.
So people living in the Global South, or you have to go to the Global South to, believe it or not, evaluate a project, or maybe you just live there right now, I highly recommend two products to protect yourself from the sun in hot and humid climates that are mostly going on in the Global South. Sunglasses you can get at akila .la, A -K -I -L -A dot L -A. You can get prescription lenses also, all made from 50 % recycled material. You get a 10 % discount
if you use my link in the episode notes. I will get a 15 % commission when you purchase any product using that promotional link. So you have your sunglasses to protect you from the sun and the global south, but you need the right clothes to be comfortable in that hot. and human climate. So I would recommend my second product, and that's men's polos, tees, and button -ups, but I'm quite sure women would find them acceptable too. And you can go to hypernaturalstyle .com. That's
hypernaturalstyle .com. And you get 20 % off any product when you use my promotional link, which you'll see in the episode notes. And if you do that and make a purchase, I get a 20 % commission on the total purchase you made when you use my promotional link. Just go to the link and you'll go straight to the company's website for purchase. So what is so cool about these
Hypernatural t -shirts? They have a patent pending Hypercool Jade technology that when it's hot, lowers your body temperature by three to five degrees. They're also breathable thanks to Supina cotton. And they are sold in premium retailers like Nordstrom. And were voted the best men's polo shirts in 2024 in Men's Journal, Forbes, and Esquire. So check those out. And now we're going to get into the Performance Measurement
Framework, PMF. And we'll say, let's start off with, you have an idea of the target groups, the services they receive. So let's go through the various outcomes. So what you're going to discover first to begin with is there are 29 outcome indicators. determining whether or not 11 expected outcomes in this $4 ,950 ,000 project were actually achieved. So we're going to go through those 29 outcome indicators for those 11 expected outcomes, and then we'll have a conclusion
as to how many of those were achieved. I should also point out to you that Humber Polytechnic offers a postgraduate diploma program in public administration, which I had the opportunity about, what is it, 2008. I taught two courses there, and one of them was supposed service quality. But now they finally have realized how bad that
was. That's what you get when an MBA is running the school of business and doesn't understand that when you're evaluating human services, it's more than just asking the human being after they got the service, whether they have been customer satisfied with the service they received. There's a whole field of evaluation and psychometrics that you have to be involved with, as you've noticed from my podcast. But now they have improved. the postgraduate Humber Polytechnic program,
diploma program in public administration. They have two new courses instead, which get it right. They have one in project design and one in project evaluation. And the first one, project design, guess what? They teach the students how to design a performance measurement framework, how to properly design an outcome indicator so it reliably... and validly measures the outcome that the project is trying to achieve. Why do I bring it up here?
Well, when I finish my critique, I'm going to send it to them and suggest that they share it with the instructor for that course. It's BUS5014, Project Design and Planning, so that maybe their instructor will incorporate it into the curriculum as an example. A real case study of not any project, but actually a project that the Humber Polytechnic is doing itself. So it's relevant because they're
going to Humber Polytechnic to study this. So you would think it would be logical and helpful for them to incorporate this performance measurement framework as well as my critique. But of course, they're free to disagree with my critique. And hopefully we'll come on the podcast and respond to what I'm going to tell you right now as I go through the 29 outcome indicators. So let's start with the first outcome. And this is the Bhutan Education and Skills Training Project
delivered by Humber Polytechnic. So increased opportunities for paid employment for technical vocation. Graduates, especially women and people with disabilities, that's the expected outcome. They have three indicators to see if they've actually achieved that increased opportunity for paid employment. The first one is the percent of targeted students who access a career service and by field of study. Yes, they can record the percent of students who actually accessed a service
who were women and or had a disability. If you look at their PMF, their quote analysis of records at the TVEN Institute needs to add analysis of records using standardized criteria of what is needed to get a yes, this student did access a career service. Right now, we do not know if the field officer responsible for this analysis is in a conflict of interest employed by the
project. So that's a problem right now. If you look at the PMF, there's somebody inside the project being paid to tick off the box that says,
yep, you've got access to a career service. Ideally, what they should be doing instead is bringing in an external evaluator who by themselves independently figures out, if I go into that institute, vocational institute, training institute, what checklist do they have to meet for me to say yeah they've got a career service going and yes the students actually accessed it so that would be better so I don't give that one a plus for the indicator I just think it's got a conflict of interest
and we don't know that it's quite simply that the other thing I pointed out is instead of looking at the records of the institute why not ask the students directly did you get access To a career service, yes or no? I think that would be a better data source than the records of the institute, which they currently have in the PMF. So that's a no -no. The second indicator was percent of industry TVET partnerships that are active. But remember, the outcome is increased opportunities
for paid employment for TVET graduates. Paid employment, not just activity, right? So here I had a beef with that. What about the percent of total partnerships that actually have paid apprenticeships? What do they mean by active? Remember, the outcome statement is increased opportunities for paid employment. Current data source is just reviewing the meeting minutes, if you go to the PMF, between the TVET institution
and the company. That's not good enough, since the expected outcome is paid employment for the TVET graduate. So that's not very good. You don't want to just look at the minutes. You want to ask even the students directly, are you getting a paid apprenticeship? That would be a better measure, a better indicator for whether or not they've actually increased opportunities for
paid employment for TVET graduates, right? So that's a no -no. The third indicator is percent of TVET institutions that have a career service specific to women and people with disabilities. So it's very similar to the first indicator, but they're now being more precise with it's to women and people with disabilities. Same problem that I mentioned previously, right? They've got to ask the women and the people with disabilities directly. That would be better. So that's not
very good. So the next outcome is increased. Whoops, excuse me. Yes. They have a career service specific to women and people with disabilities or they do not. So again, the same problem. There's a bias. They got the field officer doing it instead of an independent external evaluator. So what I've already repeating myself here, paid employment for graduates suggests a better indicator would be the percent of graduates who got paid employment, not the percent of TVN institutions that have
a career service for the target groups. Now, you could argue they need to have the career service first before they get the paid apprenticeship. Yes, maybe, possibly. But that's not what the outcome is saying. The outcome is clear. Increased opportunities for paid employment. So cut to the chase. Go straight to that. Find out how many of the graduates, percent, actually got paid employment or paid apprenticeships. That
would be more valid measure of the outcome. The next outcome is improved capacity of TVET students, especially women and people with disabilities, to engage in entrepreneurial activities, including in green business. And they have, let me see, three indicators for that outcome. Let's look at the first one. Percent of targeted students who have developed a business plan. And I've mentioned this in previous episodes with other
organizations. Same problem. Even if the percent increases, the quality of the business plan could be poor. Remember, the outcome is improved capacity of students to engage. Well, okay, they develop a business plan. You could argue that's sufficient engagement, but really, no. They need to produce a good one. That's why... They have this technical vocational institute to train them on that, right? So the PMF indicates no criteria on what is considered
a good business plan. Remember, the outcome is improved capacity of these students to engage in entrepreneurial activity. So they produce a business plan. which is improved capacity compared to no business plan at all. But those business plans should be checked on meeting a minimum level of quality. Second, there is again a conflict of interest. As the M &E officer paid by the project, not an external evaluator, is responsible
for getting this percent. Third, this percent is only measured once per year and not until year two, and not until year three. Too little. too late if they want to show improved capacity due to the project training rather than other factors. The second indicator for improved capacity of TVET students to engage is percent of graduates who identify a high level of satisfaction with the quality of entrepreneurship support programs
provided by TVET institutions. Again, if you're asking the graduates, So satisfaction has got nothing to do with improved capacity. Sure, they might be satisfied, but we're not measuring that in the outcome, right? We want to know if they technically are able to produce, among other things, a good business plan, right? So I would take this out. Or improved capacity to engage. When they say capacity, I see that as technical capacity to produce a business plan. or whatever,
right? So I would take satisfaction out. Of course, it's a good indicator telling people maybe there's something wrong with the program, but it's not related to the outcome. Third indicator is percent of student business plans that focus on green business. Okay, again, same problem as above. PMF does not mention any criteria as to what constitutes, quote, a good green business plan of high quality. Now, perhaps this is assumed, but we do not know. If it does exist, that should
be mentioned in the PMF. Again, not a valid indicator due to conflict of interest being measured by someone within the project. It should be an external evaluator instead. And again, too little, too late, and only measure it once per year and only starting in year three of the project. Now, they could argue they're not doing business plans until year three of the project. Good. Fair enough.
One could argue that even an increase in the percent of students producing low -quality green business plans is better than no business plans at all. I suggest for $4 .9 million in training, these students on business plans, we should at least measure and evaluate whether these green business plans meet minimum quality standards. So that's not a good indicator. You can't just
say, Focus on green business. We've got to see if it's good, good quality, using some external criteria of what constitutes a good green business plan. Next outcome, increased effectiveness of TVET partner institutions to deliver inclusive, industry -relevant, environmentally responsive skills training programs to students. First indicator here for this outcome is percent of TVET instructors who apply at least one named teaching technique learned through project activities. It's a great
indicator, but the same problem. Got the wrong person measuring it. The PMS shows that the project M &E staff will be observing the TVET instructors to see if they are applying these teaching techniques.
This is a conflict of interest. They mention a survey in the PMF, but it is not clear if the TVET instructors get the survey to complete themselves, which would be unacceptable due to bias, or if they actually surveyed the students to ask them if their instructor delivered one of these teaching techniques. And again, it is only done in year four and year five. That sounds a bit late. for
this project. But maybe that's only those years that the students receive this, quote, industry -relevant, environmentally responsive skills training, the students receiving it. So either way, I wouldn't accept this indicator because the data source is wrong. An external evaluator should be observing these TVET instructors to see if they are applying these teaching techniques, preferably using a pre -developed checklist before they enter the classroom to observe the TVET
instructor, right? Next indicator for this outcome is percent of TVET instructors who receive positive course evaluations. Now, we have to assume that positive evaluation is equivalent to deliver relevant, inclusive, environmentally responsible, responsive skills training. The data source is TVET institution records. I guess the course
evaluations completed by the students. The design of these course evaluations for the students to complete should be completed and delivered to the students by an external evaluator, not the Project M &E staff, which again is a conflict of interest. So you'll want to make sure the course evaluation form is specific to student ratings of the instructor. specific to delivering, quote, relevant, inclusive, environmentally responsive skills training. So that should be put in the
PMF. It's not there. We don't know. Could just be the M &E officer in the project ticking off a box and saying, oh yeah, it's positive. We don't know. So it should be more explicit. And again, an external evaluator should be doing that, not the project M &E officer. Because there's a potential for bias. So just eliminate it, right? Next outcome, strengthened capacity of targeted TVET partner institutions to develop gender responsive programs that meet economic and environmental
needs. So here, first indicator for that outcome, percent of courses developed through that industry partners confirm as relevant to emerging economic sectors. From the indicator, it is clear that it will be up to these industry partners. to declare and confirm that each course indeed has met some sort of criteria for them to conclude, quote, that course is relevant to emerging economic
sectors. The problem is that the PMF shows the data source as, quote, meeting records with industry partners and the Project M &E staff, conflict of interest, reviewing those meeting records to determine if the course is indeed, quote, relevant. This is not acceptable, right? for those two reasons. So instead, an external evaluator should survey the industry partners to get either their prior approval before the course is delivered, that it is relevant, they can review the course
syllabus. It says in the indicator, quote, developed, not yet delivered. So that would seem to be the right approach. They're developing these courses to make sure they are gender responsive. Right? So they need to survey the industry people, say, is this relevant? Is it gender responsive? And the external evaluator needs to come up with a questionnaire and criteria so that they can conclude after that that the industry partners do indeed confirm that the course is relevant.
The second... indicator is percent of courses developed that incorporate quote a gender -related concept strategy or skill so again the data source course curriculum and its review is good but again we've got the same problem we mentioned before conflict of interest right project M &E staff are reviewing it which is a conflict of interest also No mention of pre -established criteria to be used when reviewing that course content to conclude whether course content, quote,
incorporates that gender concept. So what would be better is getting an external evaluator to do the review as well as develop the criteria to be used for that review. And another indicator for this outcome, percent of courses developed that incorporate an environmental. sustainability concept, strategy, or skill. Same problem as
before. They need to get an external evaluator in there to do the review as well as to develop the criteria to be used for that review to lead to the conclusion of whether course content incorporates, quote, environmental sustainability concept, right? So you can get the pattern here. There's conflicts of interest going on with some of these
indicators, which is not acceptable. So next outcome, increased effectiveness of national TVET agencies and TVET institutions to manage a reformed gender and disability inclusive and environmentally responsive TVET system. So the first indicator is level of confidence among TVET officials in their ability to manage a reformed TVET system that is gender sensitive, disability
inclusive. and environmentally responsible. So, technical ability to manage an increased effectiveness of an agency or institution cannot be measured by TVET managers self -reporting their confidence in their ability to manage. This self -reporting is biased and fails to measure the technical ability to manage. In addition, they collected this self -reporting through focus groups and surveys. A waste of money, right? And what could
they do instead? Well, way back before any of you were born, including myself, way back in the 1950s, the Educational Testing Service developed the in -basket test. It's a test and it still exists today, I'm guessing. It's an example of where you can test the competency of managers, where they have to prioritize tasks they're given. defer it to somebody else, do it right away, make somebody else do it, delegate. That's just
an example. But if you're going to train people on how to manage, especially with this specific content, gender and disability inclusiveness, environmental responsibility, you need to design a test that can test these managers if they're competent to do that, rather than just asking them. their confidence level, right? So that's just an example. Now, after using that objective measure, you find the percent of managers isn't
going up. Then, and only then, you could hold a focus group and ask them, you guys in this room, the percent has barely gone up. What's going on? What's the problem with your ability to manage? Why was there no percent increase in those who passed? on the in -basket test. That could be an example, right? So that indicator is not very good. The next indicator for this outcome was percent of revised courses that are
initiated within their designated timeline. So initiating a revised course, perhaps to address low confidence reported in their ability to manage, I don't know, is still an output rather than an expected outcome. This is not related to the expected outcome of improving technical ability to manage. Now, I suppose you could say it's the step before, right? They've learned their lesson. They know they're not managing. And so they have to turn around and revise the course.
So that could be argued that, you know, if the percent goes up and they do that, that's a measure, again, of technical ability. But then somebody could argue. Well, we've seen the revised course and it's crap. But even if it's not crap and it's really good, what is the outcome again in this? It's to manage a system. It's a management skill they're looking at here. And maybe that's a measure of management is producing a high quality revised course. So maybe again, they need an
external evaluator to come in and say. This is what these managers have to reach if they're going to revise these courses. And I haven't checked the data source on this, but I'm assuming it's not good. I think I'd have to check, but I'm going to assume same problem. They don't have an evaluator coming in to look at those courses and say, yep, these have improved with respect to the outcome. Therefore, the management skill has improved, but we don't know at this
point. Another outcome, enhanced knowledge and attitudes about TVET as a viable education and employment pathway that is inclusive of women, people with disabilities, and vulnerable populations. So, first indicator for this outcome, increased level of support among community members for TVET education. If you look at the performance measurement framework, a sample is taken from the community members at large to see if the percent supporting TVET as a viable pathway has
increased over time. This is great. It's a great indicator. Problem is, there's no comparison or control groups, so they cannot support their claim that their project was responsible for any observed increase in the percent. In addition, there is no mention of a one -tailed, right -tailed hypothesis test using the sampling Z distribution of a simple single proportion. So any observed increase in the percent over time could fail to be statistically significant with a p -value
less than 0 .05, right? So either way, they've got a problem. At the very minimum, they should be doing what any second year undergraduate social
science major will tell you. They should be comparing the percent sample of the population that they've just done the community members sampling to see if the percent has gone up enough to reach a critical value where they can reject the null and go with the alternate and say yeah the percent has gone up and it's been statistically significant and we're at least 95 percent confident that there is that observed increase is statistically significant otherwise why do the project right
If you don't do the project and the percent still goes up, why spend almost $5 million? You got to show it's statistically significant. It's effective. Or some people say, not only effective, it has impact, right? So that's a problem. Next indicator, percent of women and people with disabilities from sample communities who view TVET as a viable education pathway. Now, From the PMF, a sample is taken from women and people with disabilities to see if the percent supporting TVET as a viable
pathway has increased over time. Again, great. But again, no comparison groups, no control groups. They still need to do a statistical test to see if that percent over time has statistically significantly increased. Otherwise, there's no point. They got to prove that the project has made a statistically significant difference. Right? And ideally, you want to have a comparison group or a control group. Now, they could be delivering this project to every single TVAN institution in the country.
And maybe they are. That's fine. Okay, so no point in having a control group. But they still have to show that the percentages they've got have statistically increased significantly. Otherwise, they should shut the program down. Another indicator is the percent of TVET institutions that implement new recruitment and counseling strategies and systems. Now, again, the outcome there is, sorry, I went to, there's a new outcome. So let me repeat that. There's a new outcome here for that indicator.
It is strengthened capacity of TVET institutions to recruit and counsel students, especially youth, women, and people with disabilities. So the first indicator for that outcome is percent of TVET institutions that implement new recruitment and counseling strategies and systems. Again, great indicator, but wrong people to see if this percent has increased. The PMF shows that individuals representing these TVET institutions will be asked, did you implement new strategies to recruit
youth, women, people with disabilities? Yes or no? This is done through a survey and interviews. Of course, they will say yes due to self -reporting social desirability bias. So they shouldn't be involved in this. Again, if you look at the PMF, it's the institutions, excuse me, serving. Of course, they will say yes due to self -reporting. So I have to look again at the PMF, but it's not mentioning as the data collector, person responsible is being. a external evaluator, which
is what you should be doing instead. Going into the TVET institution using a predetermined checklist that the institution actually has met the criteria of, yes, this TVET has indeed implemented a recruitment system. And even if they did, does the institution have a system in place to counsel these three groups? So external evaluator is what you need
to do instead. The second indicator for this outcome is level of confidence among counseling staff at TVAN institutions in their ability to effectively counsel students, including women, youth, and people with disabilities. Again, as you know, you can't do that, especially when you explicitly state ability to effectively counsel. I mean, wow. You don't ask them how they're feeling. You do an external evaluation. of their ability to counsel students. There's a whole pile of
psychometric tools out there to measure. I mean, just go to any university in Canada that has a master's program in counseling psychology. I'm quite sure there's a whole bunch of external measures, observation of their counseling with the student, rather than asking the counselor themselves to report on their self -confidence, right? So get an external evaluator, not a good
indicator. And the last one, same problem. Level of confidence among recruitment staff at TVET institutions in their ability to effectively recruit. So that's the other skill. Counseling, now they're looking at recruitment skills. Again, same problem. Self -reporting confidence levels is not a valid measure of whether recruitment staff have the ability to effectively recruit
students. So that's not good. Next. Expected outcome, enhanced access to economic opportunities in the labour market for TVET graduates, especially women and people with disabilities in high opportunity sectors. First indicator, percent of graduates who complete entrepreneurial training, who start their own business. Great indicator, great measurement
through tracking. of graduates if you see their pmf it's called a tracers study but all that money to track these graduates just to ask hey dude have you got a business yes or no wouldn't you want to ask hey are you making any money show me the money so a better indicator would be the mean monthly income earned from that business and track it over time to see if it has increased also no need to have a focus group as they talk about in the performance measurement framework,
at least not at this point, until you first use this income indicator. Then if income has not increased, you could hold a focus group and ask them, what's going on, dudes? You've started a business. It's great, but you're not making any money from it. So what do we need to do to identify what are the problems as to why? you're not making any money. Is it because they didn't get the labor market analysis right in the first
place? There's thousands of seamstresses already out there in the market that we can't compete with, even though I went to this institute and learned how to sew and graduate. I can't get any income. Maybe the United States or Canada are dumping used clothing into the market, which I can't compete against. Whatever, right? So a better valid indicator of enhanced access would not be just starting up a business, but more
money, right? And the next indicator, level of satisfaction among graduates with their economic opportunities. Again, they might not be satisfied. It tells you, yeah, there's something wrong. But what you really want to do is focus on the income, right? And of course, we're assuming. They've acquired the necessary skill levels.
I can't see anywhere where there's a certification testing program in the institute to make sure they've met a minimum standard to say that they're a carpenter or a plumber or a mason or a seamstress. Right? Or tailor, right? So if we track the income, we don't need the satisfaction levels. And remember, not other income sources, right? Got to make sure that they're actually making money from the training that this $5 million project gave them in a particular technical vocation, right?
Got to be very clear about that. It's a big problem in Africa where they'll have multiple sources of income and then you just ask them and they give you a summary number and you think, wait a sec, didn't we train you how to be a welder? How much money do you make from welding? Nothing. Okay. And you'll notice the PMF does indicate that the project training will align with, quote,
labor market needs. So satisfaction should be related to income rather than the frustration of not getting demand for their services after graduation, right? So the next indicator is percent of industry partners that hire a TVET graduate in the last year by field of study. and economic sector. Again, great indicator, but even better would be income levels. Because they could be hired, but paid a really bad wage, right? Or nothing at all. They could be an unpaid intern.
We don't know what they mean by hired. So cut to the chase, find out the incomes. And we need to show that the income levels of the graduates are higher than those who did not get trained in technical vocation. Perhaps people that taught themselves to be a carpenter. Plumber or seamstress, right? Or tailor. Next outcome is strengthened effectiveness of TVET agencies and institutions to provide inclusive and environmentally responsive TVET education to trainees, especially women
in high opportunity sectors. First indicator, percent of students who enter a TVET institution who graduate. Now remember, it's strengthened effectiveness. of TVET agencies and institutions to provide inclusive and environmentally responsive TVET education to trainees. It's very precise, but the first indicator has got nothing to do with that. It's just saying whether or not they graduate or not. We don't want to know that.
We want to know they not only graduated, but they are familiar with inclusive and environmentally responsive education, particularly women, right? And also, if you read the outcome, it's not about the student. It's about the institution being inclusive and environmentally responsive in their education to the students. Right. So I would take it out. Who cares what the percent is of students who graduate? It's completely unrelated to the outcome. That's gone. Now, the next one
indicator for this is. level of satisfaction among TVET graduates with the quality of their TVET education. That's a little closer. It's a good indicator since it asks the student if they are satisfied that the institutions indeed are delivering training that is, quote, inclusive and environmentally responsive. So you have to make sure that that satisfaction survey questionnaire that goes to the students contains questions
specific. to that expected outcome regarding what we just said inclusive and environmentally responsive the only issue here is that this tracer study is delivered again by the project m e staff rather than an external evaluator ideally you want an external evaluator to be hired to review the course content develop questions for the students that measure satisfaction levels specific to being, quote, inclusive and environmentally responsive. But on the whole, it's a good indicator.
So I've skipped the other indicator because I don't think it's relevant. So just one. So here's a new outcome. Increased participation of learners, particularly among youth, women, and learners in vulnerable situations in TVET education in high -opportunity sectors. First outcome indicator for this outcome is number of enrolled students
annually in public TVET institutions. So here the only issue is whether this project is necessary to achieve increased enrollment in TVET institutions and increased percent within this enrollment of those who are youth, women, rural, or people with disabilities. What about another region? within the country where this project is not operating, and that regional government has a policy in place with the same expected outcomes.
Now, as I mentioned earlier, if this project covers the entire country, then this is not important. A review of the project browser indicates a countrywide project reaching all TVET institutions. That's what I'm assuming. So that's a good indicator. It's a good indicator. You want to... you want to get those enrollments up, right? And they've done some community announcements to try to get that enrollment up. Now, the next one is percent of graduates who are employed in their field
within six months of graduation. Again, percent of graduates employed in their field is good, but an even better indicator would be their actual income from that employment. But I'm still accepting this indicator as a good one. And the disaggregation is really good. because you want to see those percents for women, people with disabilities, since that's a focus of the project. But again,
there is no control or comparison group. So if those percents go up, they need to show that they're going up statistically significantly, right? So they need to take a sample of those graduates at different points in time, find out what percent of the sample actually say yeah. I got employed within six months of graduation. It's a simple yes. And compare that over time using a one -tailed, right -tailed alternative hypothesis with a P value less than 0 .05, right?
The second indicator here is level of satisfaction within the industry of the ability of TVET graduates to fill labor market needs. in an environmentally responsive manner. We are assuming that the industry people who are interviewed support environmentally responsive manner. If the assumption is correct, the survey used here in the PMF would be acceptable, and this indicator complements the claim. The graduates are indeed being environmentally responsive.
Again, an external evaluator, not the project staff, would be better for designing and delivering. the survey questionnaire. But I still went ahead and said this is a good indicator, despite that. The other indicator is, finally, level of community support for TVET as a means to promote women's equality. Because, you know, some people may dispute whether women should be involved in technical locations, right? And again, the indicator here
is a percent surveyed from the PMF. And the percent could increase, but not enough to support effectiveness at P less than 0 .05. So there's no control group, no comparison group. So at the very least, they've got to do a proper sampling, two different points
in time, and do a proper statistical test. And any undergraduate student majoring in the social sciences will learn the importance of inferential statistics and showing that any increase in the percent needs to be statistically significant to make the claim that it has increased not just due to chance, right? So my summary from all of this, the 29 outcome indicators for the 11 expected outcomes, is that only four of the 29 outcome indicators are valid measures of the
project's 11 expected outcomes. So only two out of the 11, project outcomes could Humber Polytechnic claim that they've achieved their outcomes. So thank you for listening. That's the end of my review. And like I said, if you want to get the PMF and my critique in writing, you can just email me at evaluatecanadaid at gmail .com and I'll be happy to send it to you. Feel free to make any comments in the Spotify platform. where you now can make comments on any of the episodes.
And stay tuned. My next episode is going to deal with an international organization that has finally decided to publish its PMF on its website, along with the data for their percent indicators. Unfortunately, in Canada, as you can see, we don't have any PMFs on any of these. organizations like Humber Polytechnic that are funded by the government of Canada. It should be a requirement that their PMF is on their website along with the data that
goes with it. If you go to the project browser, they have data there, but it's absolutely useless data because it's not related to their outcomes or their outcome indicators. This is why I have to go and ask for the PMF through an access to information request. Thank you for listening. Stay tuned for the next episode. Bye for now.