Welcome to the Improving Development Evaluation Podcast. I'm your host, David Wand, and in this episode, we're going to revisit the Global Community Engagement Resilience Fund. You can learn more about the Global Community Engagement Resilience Fund, GSURF, at gsurf .org. You might be wondering why I am returning to this particular international development organization. because I featured
them on two previous episodes. The reason is I've discovered they've actually improved development evaluation in some ways, and in some ways they haven't, but it's a step in the right direction. And you may have figured out through my LinkedIn profile that I used to be at GSurf back in April of 2020. So that's more than six years ago. Back then, they were not in the mood. to do what I've been trying to get organizations to do when it
comes to improving development evaluation. That is putting on their website their performance measurement framework, or as the Americans call it, the activity monitoring evaluation learning.
plan or as other people call them results frameworks these are documents that list the outcomes they expect to achieve along with the outcome indicators they expect to use to measure hopefully validly the outcome statements that they're claiming they want to achieve also the data that's associated with those outcome indicators and finally the statistical analysis that shows that if the outcome indicator is indeed a valid measure of the outcome and if for example the percentage on that outcome
indicator has indeed increased over time from baseline to midline that that increase in percent is actually due to statistical significance if they're using samples rather than due to chance that's what we're trying to get proper evaluation of the project so that the claims that g -surf keeps making on its website and also on its podcast, actually have been achieved because of G -Surf and not due to chance. So far, they have failed
to do that. So just in retrospect, you may recall on one episode I showed they received $5 million from the United States government. I received two letters from the United States State Department and the United States Agency for International Development saying, We have no activity mail plan. We have no outcome indicators for that $5 million that has been spent. Bye -bye. So that's a problem. Number two, they received $1 .1 million from the Canadian government. Again,
I asked for the results framework. It took me about a year. I finally got a letter. what's more like a report from GSERF. And in that report, there was absolutely no mentioning of outcome indicators showing any evidence that they had achieved any of the four outcomes that they keep claiming they're achieving on their website.
Those four outcomes are one, increased social cohesion, two, increased ability to mobilize, organize, represent their own interests, three, increased vocational skills, followed by an increase. in livelihood, such as an increase in income. And four, increased confidence or increased critical thinking skills or increased life skills or increased self -worth or increased resilience, what they like to call sense of purpose. But finally, this episode, we are going to celebrate the fact that
GSurf has moved on. And despite their resistance over six years ago, when I was with them, they
were refusing to even mention. a results framework being on their website with indicators and they were also dismissing trying to measure all of the social psychological constructs that they are claiming they're achieving like vocational skills increases social cohesion increases attitude increases in the right direction or perception increases they were not interested at all and one of the reasons they were not interested at all, was because most of them there at GSERF
have the wrong degrees. They don't have an undergraduate degree in psychology, which is all you need to do these evaluation of these social psychology and clinical psychology constructs. And all you need to do is complete that second year undergraduate university course in inferential statistics, and you'd be off and running. But most of them, unfortunately, at GSERF have PhDs and masters in international relations or international affairs.
And I've met students who have master's degrees in international affairs and relations, even undergraduate degrees in international relations or affairs. And they never have to take a course in inferential statistics. Why? Because they're often looking at macro level indicators like trade statistics or whatever. So there's a mismatch.
That is one of the many reasons as to why GSURF was resistant to looking at these basic skills that you need to properly evaluate the services that they're delivering, where they're all focusing on changing individuals and groups of individuals in the four outcomes that they clearly state
on their website. But now they have finally, on their website, released and published some results and impact reports so I'm going to look at one of them from Ghana and go through it and give you an idea of how far they've come at least they've published it on the website which is a pioneering effort compared to a lot of NGOs and number two they've actually published the data on their outcome indicators so we're going to look at whether or not the outcome indicators
are indeed valid measures of the outcome? And number two, even if they are, the data that they provide, have they gone to the proper level of
statistical analysis? That is, have they gone back to their notes in second year university where they took that required inferential statistics course and figured out this is what I should do to make the claim that the percentage increase that I'm claiming has actually gone up due to to statistical significance rather than due to chance, that is understanding the basic second year concept of sampling distributions, then
we're in a good phase. So before I do that, if you are living in the global south or you, God forbid, have to go to the global south to Allah forbid, monitor and evaluate. an actual international development project in the field, you need a few things. So I'm not going to be talking about desiccated liver or granola. I'm going to talk about products that actually are related to going to the global south or living in the global south, where most of the time it's hot and humid, or
if it's not, it's raining. You get the idea. So sunglasses, check out Aquila. A -K -I -L -A dot L -A, where you can get sunglasses, including prescription lenses, all made from 50 % recycled material. You get a 10 % discount if you use my code, WANDCOOLT -SHIRTS, or if you use the actual link in the episode notes, which is mydeals .page backslash 1HJL. You'll see it in the episode notes. And I get a 15 % commission when you purchase
any product using that My Deals link. So you have your sunglasses to protect you from the sun in the global south, but you need the right clothes to be comfortable in that hot and humid climate. So just to remind you, AKILA .LA. That's the website. But better just to click on the My Deals. Link in the episode notes. The second product I've got is men's polos, tees, and button ups at hypernaturalstyle .com. You get 20 % off any product when you use my, my deals link or
the promo code wand, cool t -shirts. And I get a 20 % commission of the total purchase you make when you use that link. And we. Want to know, so what is so cool about Hypernatural products? They have a patent pending Hypercool Jade technology that when it's hot, lowers your body temperature by three to five degrees. They're also breathable thanks to Supina cotton and they are sold in premium retailers like Nordstrom and were voted the best men's polo shirt in 2024 in Men's Journal,
Forbes and Esquire. And I've got a new product. which is if you go to terrain .org, that's T -O -R -R -A -I -N .org, they have backpacks, totes, wallets, and bags. But what's really cool about them is, again, they're developed for the global south. They're durable, lightweight, water -resistant, perfect for travel and outdoor use. If you are in the global south, in the field, monitoring and evaluating a project. All Terrain products are crafted from upcycled feed and rice
bags sourced globally. Each bag's interior is lined with recycled plastic bottles, handcrafted by artisans supporting fair trade. And by purchasing a Terrain bag, customers contribute to global sustainability, which is what international development is all about. You get a 15 % off any product if you use my link that is in the episode notes. Again, mydeals .page backslash 1HJL. I get a 15 % commission of the total cart purchase. That's Terrain. Look at this Ghana report that's called,
if you go to the gsurf .org website, results paper. Ghana emergency grant. Now to be fair if you go to that grant it'll say it's an abridged version of a larger report. Unfortunately I couldn't find that larger report so maybe some of my comments that I'm going to raise right now are in that larger report. I do not know. I'm assuming they are not in there because you would certainly think for what they post on their website if they knew their statistics they would indicate
what I'm going to describe. in this abridged shorter version report. But I'm so happy that at least they've moved from trying to hide their performance measurement framework to a point where they're putting parts of it in this report on their website. So there's four sort of areas where there's a weakness in this Ghana report. First of all, some of their outcome indicators, just like in other episodes with other NGOs that
I've talked about, there's a poor design. where the indicator really doesn't validly measure the expected outcome. Number two, they don't mention in this report just the title of the psychometric tool that they're using to measure the outcome indicator. And I'm going to suggest some possible psychometric tools that they could use, reminding them. that if they had an undergraduate degree in psychology with that second year required course in inferential statistics, this would
not be an issue. And number three, there are some missing outcome indicators on skills training, but again, they might be in the larger report,
to be fair to them. Number four, and this is the new part, which I commend GSERF for at least putting the data on their website so we can at least have a... discussion and that is their failure to show that the percent increases that they've shown on certain indicators is actually quote statistically significant rather than due to chance and that's a basic concept that you learn in second year undergraduate social sciences or business because you have to take that stats
course so if you're majoring in psychology sociology political science or economics That's why you take that course, because later on in the literature, you're going to see p -value. And when I say p -value, I'm not talking about the quantity of urine in a bottle, right? So let's take a look. Poor design of outcome indicators. So one of the things they look at in their report is they talk about improved livelihood and employment. That's the expected outcome. for women. They
got 200 women they list in the project. And they have the indicator for that as being the percent of women from host and refugee communities with, here's the problem, quote, sufficient productive assets with access to economic opportunities. That's a problem. They've got two variables in there, sufficient productive asset and access to economic opportunities. Anybody who knows how to properly design an indicator, you don't throw in two variables within one indicator.
That's the first problem. Second problem is, as they've already stated on their website, and I've quoted it earlier, they are responsible for developing a theory of change. In this report, they indicate clearly that they are going to be training women in tailoring, parboiling rice, and shea butter processing. Training them on that. That's the theory of change. If we train them in that, we've already figured out they're going to earn more income from those vocational
skills they've acquired. They've also figured out, hopefully by their theory of change, that there is adequate demand for tailoring services, parboiled rice, and shea butter. They've already figured that out. I lived in Ghana for three years. You have to be aware they're experiencing importations in rice from Thailand and the United States. Maybe there isn't a market demand for their parboiled rice. Maybe there is because they're taking the imported rice, just boiling
it and then reselling it. That's fine. But they've already figured out from their theory of change, this is the training we're going to do and it's going to lead to an increase in vocational skill
followed by an increase in income. Done. indicator is too vague and too complex all they really need to do is look at the mean income of this group of 200 women before when the project started versus later on in the project they don't need this percent they should actually be tracking the income and also more importantly making sure that that income they're tracking is from tailoring par boiling rice selling it and selling shea
butter right So that's a poor indicator. They need to improve that because it's not a valid measure. The second thing related to that is they're missing, and again, it might be in the larger report, they're missing no indicators, outcome indicators, showing an increase in vocational skill due to the training. You need to test these women after they get the training to see if they can actually tailor properly, to see if they actually know how to process shea butter properly,
right? There's no measures of that short -term expected outcome first, because first they have to acquire the vocational skill. Pretty straightforward, pretty easy to show, especially when you have only 200 women that you're working with. So there's a failure there on the outcome indicator for skills training. For the youth, it's a little better, but for the women, it's missing. The third issue is... And it's a very simple one.
It's a basic one. And that is their failure to specify the psychometric tool that they're using to measure increased trust. For example, they have a perception change as an expected outcome for the youth. One of the indicators is percent of youth who trust people from other communities, ethnic backgrounds. So they don't indicate it. They may have a psychometric tool that they've designed. that the evaluation consultant was hired to design, or that they designed even before
the project started. All they have to do is indicate what that is. And what I've done is just for fun, I used Copilot and found four, well, I'll talk about three different standardized valid measures of generalized trust. The first one, generalized trust scale developed by Toshio. Yamagishi, highly reliable six -item psychometric questionnaire. It measures an individual's expectation of human benevolence and whether they view trusting others as a high risk. It is widely considered
the gold standard for cross -cultural. That's important because G -Surf is operating in different countries. Cross -cultural psychological research. Another one, it's only one question, comes from the World Values Survey, trust question. It's a single item. forced choice question used across global sociological studies. A third one is Rotor's Interpersonal Trust Scale, a classic comprehensive
25 -item tool designed by Julian Rotor. It measures a person's generalized expectancy that the verbal or written promises of others can be relied upon.
Now, another option for them is to hire a psychologist to develop a proprietary private just for g -surf measurement tool on trust because they may have a particular interest and i had recommended in a previous episode that i dealt with g -surf they could also use for example if they want to measure levels of trust over time between madrasa school students religious school students and non -religious public school students where they've held a soccer match and they want to
see if before the soccer match compared to during and after the levels of social capital or trust between those two groups has increased you can do it with the university of california irvine net uc net software package all they have to do is mention in their performance results framework like a report like this what tool did they actually use so those are the issues there the final one is failure to show that the percent increase is statistically significant based on sampling
or that the entire population was measured instead. It's not clear if you look at the Ghana report. They do use the S word. They do mention on page six, and I quote, the percentage of youth rejecting violence rose from 69 % to 74%, though these changes were not statistically significant. So that implies that they, out of the 600 youth that they mentioned, that they trained on critical thinking and digital literacy training, that the percent went up from 69 % to 74%. Okay, fair
enough. It's not statistically significant, but what would be ideal is that they actually showed the sample sizes from the 600. I mean, did they do it twice, two samples? And then you would understand they have to use a sampling distribution, one tail to the right, 95 % confidence. It takes on a Z normal distribution, the sampling distribution for two. population proportions, if you take samples once at baseline and once at midline, this is something they should basically be doing
and using in all of their reports. If they're going to go to the point of using the words statistically significant, then they have to go back to their second year undergraduate notes and say, ah, this is the test I use. And they can also do it for the mean income for the women at baseline and midline, right? There's a different test for that. You probably remember it. It takes on a T distribution. If I understand correctly, yep. I'm just doing it from memory, but you can
look it up. One tail to the right. You have to figure out your degrees of freedom. So they get the idea. So that's what they need to do next. Maybe they can go back to the reports and make it clear that they are drawing samples. The other issue that is a little confusing from the report is they're hinting here that they are doing sample proportions. But if you look at the populations, they're not too large to begin with. It's 200 women that got the training on tailoring, parboiling
rice, shea butter processing. And it's 600 youth individuals that got the training. And it's not clear if they measured the entire populations or samples. I think it's samples. And if it is, they need to show the sample sizes and do the statistical analysis. One tail. to the right, right? Get the critical value for T or Z, depending on the variable you're looking at, either a mean
or a proportion. And that would go a long way to showing that GSURF is getting slowly, after six years, to the point where they're producing data to support their claims that they've been making since 2020 with no evidence. So now they're getting to the point where they can actually make claims that the percentage increase that they observed and measured was not due to chance because of their knowledge of sampling distribution and instead was, quote, statistically significant.
So that's good. And I thank them for moving in that direction. It's great to see. So I think what I'll do is I'll look at the other reports
and do episodes. on this analysis and also their episodes they've also started a podcast and I made a comment to this effect on their first episode where again they made claims that the percentage had increased and it wasn't clear if it was an output or an outcome even if it's an output where they're trying to reach more scale it up as they say you need to show that it's statistically significant But more importantly, I'm focused on the outcomes because that's what
they like to claim on their website. So thank you for listening. And I'll be back with another episode on GSurf on another one of their reports. I'm particularly interested in their rehabilitation and reintegration, which they talk about on their first episode. And those are basic, again, psychological constructs that have been around for years. where there's all sorts of standardized measures of how do we define and measure reintegration? How
do we define and measure rehabilitation? And it'd be interesting to see if in those reports, they actually make an effort to do some statistical analysis rather than just saying the percent went up. Thank you for your time. Bye for now.