Welcome to the Improving Development Evaluation Podcast. I'm your host David Wann and welcome to episode five, part two, where we continue our discussion with the Mennonite Economic Development Associates International Development Organization and their project entitled, A New Partnership for Sustainable Impact Investing in Frontier Markets. We would like to give you one update.
Thanks to the Vice President of MEDA, they responded and said that of the $19 million, $15 million was paid back to the Government of Canada. So I guess we're going to be discussing the remaining $4 million, which is good to hear. So we have with us today an evaluation expert that we've had on our podcast before, Benjamin Serebor from Ghana, and we also have representing MEDA Yasser Dildar, who is an evaluation expert with the Mennonite Economic Development Associates.
And it's pleasure to have you both. So welcome to you both. And what we will do now is go through a statement of the outcomes. And then Benjamin and I will take turns providing one outcome indicator and our explanation about issues around that measurement of that outcome indicator. So starting with the first outcome, we're going to call them immediate outcomes.
We're going to start with immediate outcome, increased access to improved social impact measurement and reporting standards by small medium enterprises, investment firms, and the impact investment community. Benjamin, what outcome indicator for that outcome will you be looking at? Yes, so I'll be looking at the second outcome indicator. Which is? Which is the qualitative evidence of changing capacity in environmental and gender reporting.
Okay, you seem to have issues with that measurement for that outcome. Okay, so the first issue I have with this outcome is that it's self-reporting. When you are measuring immediate outcomes, sometimes you need to move away from self-reporting to demonstration or evidence of the application of the knowledge which was acquired by your beneficiary.
And so here, if you want them to report themselves, as you have here, 25 cases, 25% of case study and MSC study participants indicating strengthened capacity. So here they are indicating strengthened capacity. You as an M&E person should be able to observe and independently assess whether indeed they are doing an improvement in their capacity. And then there are several ways in which this can be done.
I mean, for instance, after the training, you can give them maybe a case study or something to work on to see whether indeed, for instance, they are now able to report on gender and environmental issues or not. So here, the only issue I have here is with the self-reporting. Now, secondly, when you look at the way the indicator itself qualitative evidence of strengthened capacity in environmental and gender reporting.
Now, indicators are supposed to be smart so that when we are reporting, it will be very much easier and then straightforward. So I made a recommendation of using an indicator which reads percentage of project beneficiaries who have demonstrated a high knowledge and skills in reporting on gender and environmental issues. So this is an indicator for example that could consider using instead of what you have here because then it makes it very smart. You can have a baseline.
I know you've done it already. And then when you have an indicator like this, you can have a baseline and then it becomes very much easy for you to measure the demonstration of knowledge. So this is what I have for the sense here. So Yasser, what would your response be to that? Yeah, I think I agree and this is how it was done because for each of the outcomes, we usually have two indicators, qualitative and quantitative.
The first one, if you look at it was like numbers or percent of SMEs reporting against the enhanced social impact measurement system. So at the baseline, they were not reporting against any social impact measurement system. And then we introduced the system and that took us a while because we had a criteria what the social impact measurement system will look like.
So we piloted that social impact measurement system in phase one and then refined that in phase two because the criteria was like the social impact measurement system should be applicable to all sorts of companies regardless of size, geography and sector. So we wanted to have this standardized system. And then once that was developed, so we built the capacity of those SMEs on how to report on this impact measurement system.
And then we asked them to send us back their reports and we looked at them and that's how we reported against this indicator and provided back data to global affairs like how many percent of SMEs reporting against enhanced social impact measurement system. Now, the qualitative evidence was just to triangulate the numbers. So we provided some individual case studies at the end of the project. We wrote like 30 different case studies just to augment the findings based on that percent of numbers.
So I agree. Yeah, so that's how we did it in actual reporting. And as you understand from the monitoring and evaluation perspective, so these indicators once you develop, they are further kind of elaborated. We call that like indicator definition sheet in the USAID world. It's the performance information reference sheet. So we define each of these terms and how they will be reported, measured and reported internally and externally because the PMF is just providing you a list of indicators.
Within indicators, you have certain terminologies that need further elaboration. What do we mean by social impact measurement? All those kind of things are further defined. And one thing, I think the overall thing that we need to keep in mind that this is a blended finance project. So the core of the blended finance project is to incentivize companies to implement social impact measurement and ESG related practices, but also to crowd in the private capital to amplify the impact.
And some of those traditional performance measurement framework might not be so relevant for the blended finance project, but we try to include all of those aspects. So we have some learnings coming out of this PMF and for the project PMF was just one reporting instrument.
But just one question is you're talking about small and medium enterprises, but it's the individuals inside of them that you're claiming you've strengthened their capacity to be able to report better on environmental and gender reporting. So it sounds like you've got some tool there where you're actually measuring a real human being inside of this small and medium enterprise. And somehow it's been aggregated up. So we missed that. So what could we get at?
It sounds like there might be some tool that you have where you actually measure a human being, because that's what we're trying to get at here, because it's not the institution or the organization. It's inside of them. You can't just say a small medium enterprise is reporting with strength and capacity unless you measure people inside of that. And what we're saying is there's a bias because when you use most significant change, it's based on asking them how they feel they're doing on that.
What we're claiming is that you need to do something more objectively. So do you have something lower down where a human being inside that small medium enterprise is actually being measured on their ability to report on environmental and gender reporting in a better way? Yeah, we did do this. We were like evaluation conducted for this project and they did ask these SME to extend to- Well, it's not an SME. Sorry to interrupt. It's a human being inside of the SME.
Yeah, that is entities are made of human beings. So whenever, yeah. So all the entities are- Right, so you're measuring those human beings. Yeah, but we are measuring the practice of those entities. Of course, all the works are done like for instance, if we say in an organization, their organizational human resource practices are great. So it's not like the organization practices, it's the individual within those organizations implementing the practices.
So that's like, you know, all the organizations are made up of individuals and then definitely we worked with certain individuals who were the focal persons for us receiving incentives to implement social impact measurement system. And we worked definitely with them and they had increased ability to report on this, but it was the practice of those SME that- Benjamin, any comments? Yeah, I mean, Ash is very rightly mentioned.
Once we have the entity individuals mentioned here, obviously we need to measure that. I agree with yourself that yes, those individuals are going to make a certain contribution at the institutional level. And so we can measure that contribution they are making at the institutional level. I mean, or at the institution now for instance, we've participated in this training.
The institution has not instituted measures to ensure reporting on gender and environmental issues at the institutional level. But then at the individual level, so what do we do? Probably there could be other practices, but not them. So then it's always good that we look at the individuals also in addition to looking at what the institutional level is.
Yes, and just saying the number of SMEs are reporting against this system is, in my opinion, not an adequate measure to make the claim that their capacity is strengthened. So we can leave it at that, but I understand what you're saying is that the PMF is maybe at a higher aggregate level where that's just simply not the case. So that's fine. We can move on.
But I mean, I think our point is still valid that you can't make these claims of strength and capacity unless you're going to measure individuals on their strength and capacity. I mean, and that's a common problem that we have, but we won't repeat ourselves. I'll move on to the second, unless, yes, here, you want to make another comment before we move on to the next outcome. Yeah, I think it's the same like whether we were focusing on individuals or entities. So here the focus was entities.
Even those individuals leave, those SMEs will continue reporting on that enhanced social impact measurement system. So because the purpose was to institutionalize these practices. So we make this system available to those SMEs. So even those individuals who participated in initial discussions and trainings, they would leave, but SMEs will still continue reporting on the social impact measurement system.
And there may be, you may have an internal system where the percent of those SMEs that are reporting against, but it's not here, but ideally you'd want to have the percent of those SMEs that actually reach a minimum standard, like a SMART, a minimum standard of acceptable strength and capacity. And you just tick the box and say, now, based on our assessment of those individuals in that SME, we can conclude that that SME as an entity has strengthened its capacity.
And we don't see that here, but we suspect you've got it somewhere else. So we'll move on to the next outcome. Improved business knowledge and skills in relation to social and environmental issues amongst selected small, medium enterprises. And the indicator I picked there would be the number of small, medium enterprise innovation grant recipients reporting improved business knowledge and skills in relation to social and environmental issues.
And it's the same issue, basically, where you're claiming that the SME, the small, medium enterprise, is the number is reporting on improved basic knowledge and skills. But, and you do actually have an assessment. So I've actually rated it as good. The only issue there, yes, here, we believe, is that meta, your organization should not be involved in that assessment.
Ideally, you'd want to have a third party go in and do that assessment to see actually if the individuals inside of that small, medium enterprise actually have experienced improvements in their basic knowledge and skills. So we would prefer a third party.
And then the other reason is, if you look at the frequency that you're measuring that, you probably want to have a comparison group of another small, medium enterprise somewhere else in the country, because you're doing this globally, where they're not experiencing your training on this improved business knowledge and skills, where they could be figuring it out all on their own without $4 million in a project, where they're going to YouTube or talking to colleagues or other entrepreneurs.
So that's my response there. You'd want to use a third party on this assessment tool that you've developed and maybe have a comparison group where you can show that your SME that you've trained or through the assessment shown that the knowledge and skill levels of that SME is going up greater than another SME where they're not involved in the project. What would be your response to that? Yeah, I completely agree with what you said.
It's only the limitation in terms of resources and also some logistical constraints, why businesses which are not part of the project will give their data or would like to go through the assessment. So if I'm a business and I'm not benefiting in any way, I'm least likely to provide data, even those SMEs because these are businesses and I have written in one of my responses for them like reporting on social and environmental and gender aspects is something new.
They are good at financial reporting and things like that. So even for those who are participating in the project, the challenge and if we are asking those SMEs or those businesses who are not part of the project and they are not benefiting in any way, why I should be spending my time and money. So that was the constraint, practical constraint that we couldn't do the comparison group from evaluation perspective. I 100% agree with you.
And also I take your point on the third party assessment is the again, function of resources so there was that limitation. And how frequently, another option as you've probably seen from our trailer and the podcast is another option is if you can't get a comparison group, you could just measure more frequently their levels of knowledge and skill. How many in the PMF I have it, but I thought I'd just ask you how many times do you do this assessment? I think it's not very frequent.
I think it's just what three times a year, but correct me if I'm wrong. Yeah. Yeah. It was done quarterly basis. If you look at, and that's why it becomes limiting is, you know, we have like in the response that I sent to you, we have enhanced impact measurement system developed as part of the project and research was an integral component. So university of Waterloo was supporting us on that one. So they did look at some like secondary kind of businesses and tried to do the comparison.
It was not like robust, but at least there was something that was, that was done and it is not part of PMF. But I agree and take your point. Like always comparison group is always helpful. It's not just pre and post. So you need to have a counter factor. Yeah. To prove your point. Yeah. Quarterly is not bad. Okay. So let's move to the next one. Benjamin, I think you're going to do the indicator here.
The outcome is strengthened management capacity amongst investment fund managers in selected countries. And the outcome indicator you're doing there, Benjamin. Yes. So the outcome indicator I'm doing here is level of business skills and knowledge among men and women. So I think here again, because we are, we are measuring their level of knowledge and then we are looking at like a scale of 70 percent. So we are using some of the new knowledge and skills. At the higher, at four or higher.
So I mean here again, we are looking at it like self-reporting. And we, or is it not going to be good to have an independent assessment instead of the people doing the assessment themselves? Because probably there could be that assessment bias. So what usually happens is that you as the implementer establish a scale which you are going to use to measure this knowledge. And so you give them an assignment or an exam or something just to measure the knowledge that is acquired before and then after.
Now based on this criteria, you'll be able to establish whether indeed there has been an improvement in their knowledge. But when you ask them to assess it, they can say, oh yeah, we've found an improvement in knowledge, blah, blah, blah. You can, later at the intermediate outcome, you can measure it. But initially it's important for you to know the kind of knowledge they have acquired here before you get to the intermediate.
Because at the intermediate, if they did not get the right knowledge, then you will not be able to achieve your target at the intermediate level. So it's always important that you independently assess whether indeed what you taught them has acquired the knowledge. Because I remember we did the training, we did before, and then we did after assessment. And then even after assessment, some people were getting lower than what they got in the before. I mean, I was surprised that we got that result.
So it's always important that sometimes you do some of this assessment before you get to the intermediate level. So if there are any gaps in the knowledge at that level, you'll be able to correct it before you move. And then we can also look at the frequency of measuring this also, during the training or immediately after. That's what I have for this outcome. Yes, Asir. Yes, Asir, sorry. Yeah, Medneban, I agree.
So for the mentorship program, we did develop a very robust kind of assessment criteria, and that same form was used at the beginning of the mentorship program, like when a fund managers would decide to participate and towards the end and in between as well. And then we also tried to do the satisfaction with the process itself. I agree. It is self-explanatory. Well, this is it. What you're introducing is not in the PMF. We've just got both significant change and case studies, I think.
Correct me if I'm wrong. Right, Benjamin? What exactly was used for that? It says level of business skills and knowledge among mentored fund managers. So I think the fund manager, people being mentored were reporting. So that's a self-reporting bias right there. But you're talking about an independent assessment, Asir, which I don't think that's going on here, correct?
Or are you saying that's in addition to the self-reporting bias, there was an external assessment of those people being mentored? No, it was a self-reporting assessment. So we agree then. All right. Gotcha. Yeah, so it was, but it was like pre and post. So we, they were like identical. Sorry to interrupt. So let's just cut to the chase. If we've got this self-reporting bias, would you not prefer to have some external objective assessment of their business skills and knowledge?
And that the only reason you didn't do that is maybe you didn't have the money. Correct. Resources have been one of the constraints. And second part, as I have been saying, based on my experience working with the private sector, especially investment fund managers, and these relatively speaking, bigger businesses, is the challenge in getting data is not always easy.
So we have like PMF developed from the perspective of traditional social economic NGO sector or civil society or sector, but from the private sector, it's really, really tough. Even getting this self-reporting assessment was really easy. Interesting. So we need to understand that. And this is not the first project. Like we did another one. So same thing. Like we need to actually sell and make a business case by this data is important. So data should be used for like making a business case.
If they don't see a business case for providing data, so they won't do it because they are putting in their money and everything because they will be attracting private capital. And the success of this project was this project established a fund which is still functional. And they're still providing investment money to SMEs in developing countries to create jobs and was able to attract around $150 million.
It makes you wonder why even have these outcomes of human development in this project if it's just a bag of money going in and if it's actually developing what you said, crowding in of $85 million, et cetera. That means that they in theory have the capacity to do whatever they got to do. And it makes you wonder why all these human measurement performance outcomes are in here and why you had this mentoring.
Maybe you didn't need it if these SMEs and investment fund managers in these developing countries succeeded in generating all this profit and capital. Yeah, no. I think they still need like when we do mentoring. So they were paired, for instance, fund managers from those developing countries with one of the fund managers in North America or Europe, how it worked like establishing networks. So what people, you know, fund managers in those developing countries lack is the networking.
There were two or three other areas of improvement like how to pitch their ideas to attract funding. So they were successful. They benefited from this aspect of mentorship. So they, you know, they learned how to properly pitch their business ideas, how to network and expand their network. So these are all things that are important. It's interesting. You're making those claims, but based on the measurements, they're not supported yet.
I mean, if you could have an objective assessment, then you could actually support what you've just claimed right now is that they have the technical capacity. But on the other hand, you could say, who cares? Because in the end, your private capital worked. I mean, it worked. I mean, on the profit side, you got all the money that you developed.
So maybe you should give yourself a break and say, it doesn't really matter because we managed to in the long term generate the profits through the investment. But if you've got your outcomes in the PMF, all we're saying is they should be more objectively measured so you can actually support that claim that they actually did increase their capacity. Speaking of that, the next outcome is increased capacity of investment firms to serve the financial needs of small, medium enterprise clients.
And the outcome indicator for that I'll be looking at is qualitative evidence of increased capacity to serve the financial needs of the small, medium enterprise clients. And again, the issue there is the self-reporting bias through most significant change in case studies. But the other thing that's interesting is instead of asking the investment firm, another approach would be to go to the clients and ask them.
And this will come up, I think, later with Benjamin on another outcome indicator is instead of looking at the investment firms that you're trying to develop, their increased capacity is to look at the clients who got the services from them and ask them what their satisfaction levels are on the services that were delivered by your project beneficiaries.
And also again, again, focus on a more technical objective measure rather than the self-reporting What would your response be to actually going to some of the clients instead of the beneficiaries in the project? Yeah, you are right. So in this case, our beneficiaries were SMEs. So we did, it's not represented here, but we did talk to all the SMEs, like to what extent they feel supported by the investment firms.
And at one level, investment firms or funds, they were also clients because of why we have this indicator here, because eventually this project was to raise private capital and attract money from private investors and then invest in these investment funds in developing countries. So we asked them, do they have like improved ability to serve the financial needs of clients in this case SMEs. And then we also asked SME to what extent they feel supported in terms of meeting their financial needs.
But I take your point on again, on self, so there is definitely self-reporting. So it sounds like then you do have another indicator there. You actually went to the clients, the SME clients, and you asked them what do they think of the investment firms in terms of serving their needs. But interestingly enough, it's not here in the PMF. All we've got here is the investment firms doing self-reporting on whether or not they thought they had increased their capacity in serving those SME clients.
So that's interesting. Okay. So it's not there in the PMF. That's unfortunate, but it is what it is. But that's moving in the right direction at least. That's good. Yeah. So just one comment, because there are so many things which are not in the PMF, which we did as part of our measurement. And that's why I sent an email like this traditional PMF is just one component. There were like three other different approaches we used to measure the results. But I take your point.
Well, I mean, it should be in the PMF. I mean, all the indicators should be there to cover the outcomes. And I mean, that's what we're working from. It's easy to put it in. But yeah, sure. That's what happens, I guess, when the PMF is not adequate.
I understand that on the economic side, but I still think Benjamin and I are strongly concerned about the self-reporting bias when you're talking about human beings, increasing the capacity of human beings in training, because a lot of the projects, as you know, are in training. So that's really where we're hoping international development organizations will start to move in that direction. But we don't want to repeat ourselves too much here.
The next outcome is improved access to social and environmental investment services and financing for select small medium enterprises in Asia, Latin America and Africa. And again, it's the same issue. Now, I think I'm going to skip this because improved access is just an output. So, I'm going to jump and just make the note that... Just one comment on access. If you look at results-based management strategy of global affairs Canada, access is at the higher level, intermediate level.
Yeah, we can dispute that. But I mean, when I say you're just... When you say number of SMEs receiving investment from local investment funds, that's just a delivery of a bag of money. And I consider that to be an output. I'm wondering, I'm more interested in what happens after. And of course, it's improved access because you're giving them something they didn't have before. But in terms of an expected outcome, that's not adequate enough. You go home after you deliver the money.
What you want to know is after you deliver the money, what happens? And so that second part, qualitative evidence of improved access, that again, is just showing that they got the money. But what we're more interested in knowing is after that, what's the expected outcome? So that's why I moved to the next outcome where it says select small and medium enterprises and investment firms are used, quote, improved social and environmental impact measurement and reporting standards.
So in that one, what I looked at is the outcome indicator is interestingly enough, it says they use the standards. And I guess what you're doing here is if I look at the indicator and you have two here, which is interesting, you have the perception of fund managers and small and medium enterprises on user friendliness of improved measurement standards.
And you also have another indicator, number of investment funds and small and medium enterprises with improved social and environmental impact measurement and reporting standards. But the issue is, how do you know that the number reporting is actually experienced an improvement in social and environmental impact measurement and reporting standards?
Like you've got the number reporting, but ideally what we want to know is within that number is the percent who are up to the standard that you've described. So you may want to elaborate on how do you get at the conclusion that the number you are reporting has indeed reached this standard. So number by itself is really not sufficient. You need to say number and percent of that number that actually reached the standard.
Did you, for this indicator, did you actually have an assessment or something that you used to claim that the number actually equals those who've actually improved on the reporting standards? Yes. Yeah, we did like number and percent. So what we need to understand here is as part of this project, there was no social and environmental impact measurement reporting that investment firms or SMEs were using. So we came up with 22 IRS indicators. So this is this global impact investing network.
So they have developed a social and environmental impact measurement system. And this is industry wide kind of practice. So we identified 22 of those and we asked these funds and SMEs to report on those IRS indicators. So that was the standard. So it was not that these SMEs or funds were reporting on those IRS indicators before participating in the project. So they were introduced and then we would have the quarterly reporting from them on those indicators.
And along the way, so we had trainings, you know, all the coaching to them. So they can make improvement. Yes, it was done as number as well as percent.
But the issue is, and Benjamin, having lived in Ghana for three years, I'm thinking, how do we know they're just not sitting under their favorite palm tree, drinking palm wine, ticking off the box with these indicators and that they as an organization, the people inside of that organization don't have any improved ability when it comes to reporting on social environmental impact standards?
I mean, how do we know the report comes in every quarter, as you described, but how do we know within that organization if they really are knowledgeable on what we mean by social environmental impact? Yeah, we did what we call like periodic audits, which means our fund manager and then the impact and knowledge management team and ESD team would visit these funds. And at least all of them were visited maybe a few times during the life of the project.
And we actually talked to not only SMEs, but also to their customers. So, for instance, when we developed those case studies, so the case studies were not developed just talking to these people. So, I can give you an example. So, we visited a few educational institutions, talked to students, the improvement that had been made and compared it with the indicators that they reported. So, those were some of the like verification mechanisms.
So it begs the question, why not put that in as an indicator where you have the percentage of investment funds and SMEs with improved social environmental impact measurement and reporting standards based on that audit or based on audit reports? Why not change the indicator to that? And that would actually show that up to a minimum standard, X percent of those SMEs that have reached that standard and that your data source should be those audit reports, not just the number of reports that come in.
Yeah, so that's why I think at the beginning I said like we do develop indicator definitions or elaboration sheets. So, it further says, okay, reports is one source of data that we have in the PMF. In addition to that, what else needs to be improved? Yeah, I think you should change the indicator here. You should put it to the number in percent with improved as verified by the audit reports.
And then you can go to the data source, which is the audit reports, not just the reports coming in every quarter. And then you can go in and explain the data collection method, which is, you know, the audits come in, blah, blah, blah, blah, blah. And then when people read it, they can go, oh, okay, that's a better indicator that shows that your project is actually verifiably showing validly that there's an improvement. Yeah, I think that would be this aggregation is really nasty, in my opinion.
You just can't aggregate up because at the bottom, at the end of the day, it's human beings and individuals, not institutions, not SMEs, not investment firms that this money is trying to improve. Yes, there's a bag of money separate that you want to bring in and have capital growth. I get it. But on the human side, you really need to stick to, you know, basic psychometrics. Otherwise, it just gets gets lost in the mix.
Yeah, I don't necessarily agree on that one because some of these funds, they after now the project ended in 2016 or 17 still funds keep reporting and many of their staff have left who were part of the trainings. So it has been institutionalized. So it's not about individuals. It's about those entities that we were trying to improve. But still, as we just mentioned, if they produce a report, somebody's got to look at it and see if it's up to standard.
I mean, there's there's there's a connection. So you claim it's institutionalized. You've got to look at the reports they're producing and see, is it is it up to standard? Is it still up to standard? I mean, and that's where the audit, I think, comes in, where you where you have to be more precise and not just assume that the number that reports is it equals improved standards because it may not. And in particular, especially when you have turnover with staff within the firm or within the SME.
But, you know, that's the same point. You just got to you got to measure a little lower on your level because once you aggregate up, we don't know where it's coming from. So we're going to the next outcome, which is strengthened business practices for economic growth, job creation and social impact amongst investment firms and small medium enterprises. I think that's me again. So I'm thinking it's a bit redundant here.
But what I again talked about on this one was the qualitative evidence of strength and capacity. And again, the same issue. So I think what we'll do is we'll move on because it's the same issue that we've been discussing, which is trying to get more objective measurement. And we will move on to the the next outcome, which I think, Benjamin, you're doing, which is the outcome. Go ahead. Yeah, just trying to understand on 1200. So here we are talking about actual business practices.
So and those whenever we do like business performance, there are standard measure in the industry. So if you have improved business performance, either you are creating more jobs or you are more profitable, you have expanded customer base or you have like greater inventory. All those kind of things are part of business performance. And that's what we were trying to include here because number of jobs created is one of the key for this project from social side.
So we wanted to create jobs for women and men by these investing companies and of course underneath by there. So I'm trying to understand from from my own perspective, how would you measure the business performance? Well, I put in the summary. We asked the question, there is a baseline measure that measured once a year. And my question was, is that enough to show the project created the 15,000 jobs at 10 million dollars in tax revenue?
How did you show that, you know, that your project funds created this and in which countries? Yeah, we have all the disaggregation by SMEs, by funds, by sector, all these disaggregation was there. Of course, we can provide here. But so how many women, how many men, how many youth, which company, which year? So all the disaggregation. Yes, but the challenge there, and to be fair to you, it must be very difficult and an economist could do it.
But you would need to show that the the monies that you brought in, the financial contributions were statistically significantly greater for your project compared to SMEs of a similar size that did not get capital infusions. And that's a complex question to answer. Yeah, no, I agree. The comparison that is a valid comment across all the PMF indicators.
And it's something, especially with investment related projects, it's hugely difficult, not only a matter of resources, but also why other companies will participate. But that comment is well received and definitely we will try to include in our future interventions and we are trying to the extent possible. For me, it doesn't really matter. More importantly, is the human side for me, the fact that the measurement isn't up to standard.
But you could argue who cares? You created 15,000 jobs, $10 million in tax revenue. Who cares if another SME that wasn't involved in the project, you know, as long as you got your money back. And that's the other issue is you could put the 15 million you got back as part of your PMF, although that's more an efficiency issue. But you could give yourself credit for that. And that's why we're noting it at the beginning of the podcast. Let me move on to the next outcome.
Expanded provision of investment services by investment firms to small, medium enterprise clients. And Benjamin, I think you're looking at an indicator there, outcome indicator for that outcome. Yes. So I'm looking at the outcome indicator, quality, evidence of overall improvement in investment services. Yeah. Also, I think we've discussed it over and over. When you get to this point, especially on the intermediate outcome, then the focus is going to be on whoever is receiving that service.
They're talking about improvement in services. So I would like to hear from the client. And I think when you go to the ultimate outcome, then you understand this better. I'd like to hear from the client what was the standard of services they were receiving before. What is the standard of services they are receiving after the project. Let's do that comparison to see whether they have been some improvement or whether the quality of the services provided by SME have improved over time.
So this is also the issue with this indicator we have here. Yes, sir. Your response. Yeah, partially agree, because if you look at the outcome is expanded provision of investment services. One of the provision of investment services, the capital itself, and then other investment related technical assistance that was provided to funds and SMEs.
And those were not listed here in the indicator itself, but we did capture data through interviews with these SMEs and investment funds, but other kind of technical assistance actively received as part of the investment from the fund. And we don't like there was like, you know, governance practices that have been improved as a result and some other things. But probably it's not reflected in the PMF and we could have done it, I think maybe elaborated, but it was part of the indicator.
Yeah, that's unfortunate because it should be in here. If you're if you're doing technical training. You should have an indicator on their technical capacity, objectively, not through self reporting, not through most significant change not through case studies, you should actually have a third party coming in and saying yep, they're up to standard. Thanks to the training.
We can all go home now, but you got to give yourself credit if you're doing technical training support, you need to show that technically, you know, objectively rather than this qualitative evidence. And yes, I mean, I think that's why I think we've done a lot and then we don't have it in the PMF. One trick in developing this PMF is that we don't want it to be too bulky.
But then we just have to make sure that some of the key things you are going to do is reflected in the PMF, which you have mentioned, because I mean, I'm getting a little of what you have done compared to what I see in the PMF and then the kind of instruments you are using to measure everything.
So in subsequent ones, I think it will be good if you are able to elaborate, specify, I mean, some of the things in your PMF. If you make it much easier for a third person or someone outside the organization to understand what you are actually doing. What you have here now doesn't give that a language.
Yeah, well taken. Yeah, bias and there's a point where at least you're doing self reporting bias. Yes, here we've got some projects coming up on a future podcast where all they do is take attendance and everybody goes home. So you're at least at the second level where you're at least attempting. You're asking the wrong people, of course, but at least you're attempting to get something out of the people that got, you know, the services from the project. So that's good.
And we just think you should give yourself more credit than just self reporting. But, you know, we understand we get it. Sometimes you don't have the money to hire a psychologist to come in and do a proper objective, technical measurement. Yeah, and only one comment.
You know, I agree with what you're saying, Benjamin. Only thing is, we consider PMF is just one instrument, it can be improved based on what you said like always having comparison group and more objective measures, but then it's a matter of resources.
How much resources you want to put in there. But we do have for all our interventions, including this one, and this one in particular, we had a very elaborated search and learning agenda developed and which is not part of the PMF as such, but when it comes to reporting, we would always benefit from that research and learning activities.
Yeah. But again, well, you need to put it in the PMF in terms of a more valid indicator for the outcome, because you can't have it both ways. You can't claim you're achieving these outcomes of strength and capacity and then have a weak measure of that.
If you are doing a better measure somewhere else in the research and learning document, or wherever it is, bring it into the PMF, because that's where you're going to be measuring performance. So, you know, if you're doing it, then you should bring it in. That's, that's all we're really trying to get at. The last outcome.
Sorry. Yeah, on that one, I agree, but PMF is a linear kind of framework and it has its limitations. So for instance, we usually in our projects do a lot of things around gender equality, environmental sustainability and those kind of things. You can't put everything in the PMF, it will become really, really difficult to manage, there will be hundred indicators and so many other things. So, you can only do with the PMF certain things, but not everything that the project is trying to do.
Yeah, well, yeah, but in the outcomes that you have here. That's all we're saying the outcomes here are clear your statements of what you are claiming that you're achieving then you just got to make sure the indicator for that outcome is reliable and a valid measure
and what we're saying is it's not there. Yes, you could have another 50 outcomes flying around in the project but you've already made the decision that these are the outcomes and there's only nine that you've listed, all we're saying is for those nine, make sure that your outcome indicators are tight,
they're reliable, they're valid, that's all. But yeah, I get your point, you can have other outcomes flying around, but for the nine that you got here, you got to improve on the outcome indicator for measuring those nine, so that you can even show all the hard work you've done is attributable to the project.
So yeah, it's yeah, I'm just repeating myself so let's move on to the last outcome. The expansion of up to 130 socially and environmentally responsible, small and medium enterprises, serving up to 5,000,000 and 50,000 women and men includes suppliers, employees and consumers of small and medium enterprises, small and medium enterprise goods and services in emerging frontier markets. And Benjamin, you were going to look at an outcome indicator for that outcome.
Yes. So, the indicator we have here is change in select SME supply and demand. Which I believe is a good thing to measure. But then one thing usually is that how you pick the indicator, it will determine how you are going to measure it. That's why your indicators always need to be smart. It needs to be specifically measurable, achievable, realistic, time bound. Now, here I see, immediately I see it, I see that the SMEs involved in the project, we expect them to have an interest in clients.
So when you look at the discussions we've had on the immediate and intermediate outcomes, where we talked about the clients, the clients, we want you to assess what the clients are saying about the SMEs. Now they all end up here talking about why this SME is going to have more clients, probably because they are satisfied with something. That is why most of them will subscribe to work with this SME.
So this is a good indicator. But the way it has been put here, I think you can do something about it or you can work on it. So, let's say you can look at a number of clients or new clients that have subscribed to the services of the SME. So then before you implemented the project, you record the clients or customers that they have and then after the project you record the customers that they have.
So all that we have done, all the discussions that we've had on the channels, we now deal with gender and environmental issues in the services that we provide. Then how did all this contribute to increase in the client base or increase in their customers? So then once you get this indicator right, it helps you to develop tools which will enable you to measure this objective. So I agree with you that measuring the number of clients is good, but then you can look at the indicator.
And I mean one thing I also found out is that we are talking about expansion. So what expansion are we looking at? We are looking at expansion in terms of client base here. We are looking at expansion in terms of new services. We are looking at expansion in increased employment, which we have mentioned already. We are looking at expansion in terms of branches of the...all these are some of the things that you could consider.
How probably increase in profit margin or...so when you talk about expansion abroad, sometimes you need to narrow it down. Here you circled on the client as the expansion you are talking about. Probably there could be other expansions or other measures which you can use to look at how these things have expanded their operations. Yes, here? Yeah, no, I think that makes sense. Yeah, we take one major one, the client base, I think.
That's what we focused on in terms of like how they become socially and environmentally responsible. That was a function of like what kind of clients they have. So they are reaching to more women and other groups which they traditionally wouldn't do. And this was not...this is not in the PMF, but in the detailed reporting because we were receiving quarterly reports from them in terms of their client base. So that's how we reported on this outcome. But I get your point.
Benjamin, any further comment? No, I mean, the other comment I'll make is that talking to him, I get a clear understanding of their...so in future it will be good to just try to make things clear so that whoever thinks it will be able to get what we are trying to make. Okay, so this has been a very interesting discussion. And as usual at the end of these episodes, we usually make an assessment, but I think we've made our points throughout each outcome indicator that we've raised.
So we won't repeat that. But we do want to know, yes sir, if your organization would be willing to consider in the future all your performance measurement frameworks, if you'd be willing to put them on your website, both the blank ones, which we have, but also the data that as it comes in for each of your indicators.
If you'd be willing to do that, we understand, of course, the government of Canada does not require you to do that. And in fact, we're advocating that the government of Canada put on their website, all of the performance measurement frameworks for all of their projects, you know, roughly $4 billion a year. What would you be able to, I know you don't represent the organization, but would that be something that you could look at?
Yeah, definitely. I think we have some like partial PMF data in terms of impact on our website. So, but internally we had a discussion, maybe we should put PMF data on our web. There are like issues related to like data privacy and all that. We could do aggregated data, I don't see any harm in it.
If we are not providing individual SME or individual funds name, I think it is possible. So we are having internal discussions, how we could do that. So yeah, that is something up for discussion. So I can't promise anything, but partially I think we will have some impact.
Well, they say the first quality of leadership is to lead. So if you did that, you would be a pioneer, because I don't think there are many Canadian NGOs are out there putting their blank PMFs on their websites, as well as the data for the indicators from those performance measurement frameworks.
It would be great because then they could actually show that they are actually achieving their outcomes. That is assuming that they are properly designing their outcome indicators to measure reliably and validly those achievements of those outcomes.
So it is a movement in the right direction. It would be great. At the same time, at the end of this podcast, we will be sending both parts one and part two, which is today, to the Minister for International Development, as well as the International Development Shadow Critics for the Conservative Opposition and the NDP.
And we'll also be copying you on that, along with the PMF and the Excel summary sheet. And we'll be advocating for exactly that. That they, one, put the PMFs for all projects that the Government of Canada funds on a Government of Canada website, preferably the Global Affairs Canada project browser, because that's where all the projects are.
And number two, that they improve their own standards, because if you know Global Affairs Canada, their own standards on designing the indicators in a performance measurement framework is to stop doing this qualitative self-reporting bias and use objective measures.
It's in their own guidelines, but they're not following them, as you can tell from a lot of performance measurement frameworks that we review on this podcast. So that's the second point we're trying to say is improve the design of the outcome indicators and then make them public. So that's where we're at. We want to thank you for taking your time to respond to our critique of your performance measurement framework. And we look forward to maybe having you on the podcast again for another PMF.
Yeah, no, I'll be happy to do that. And thanks, David, for providing the opportunity. And thanks, Benjamin, for your comments as well. Thank you both, and stay tuned for episode six. And bye for now.