Welcome to the Improving Development Evaluation Podcast. I'm your host, David Wand. And in this episode, we're going to follow up on my previous episode and look at the Global Community Engagement Resilience Fund. And you can learn more about GSERF, G -C -E -R -F, at gserf .org. And we're going to look at the Western Balkans, which they have on their website, a results... report entitled Results from Round 1 and Round 2 Funding in the
Western Balkans 2020 to 2025. And we're going to focus again, like my podcast has been doing since 2023, identifying two major weaknesses in evaluation design. The first one is failure to do any statistical analysis of their indicators. or doing statistical analysis that is just simply incorrect. I'll be looking at that in this episode, specific to the Western Balkans report that they've put on their website, which I commend them for.
This is great to see. But more importantly, I'm going to focus on the second area, which is the failure to use, as you've noticed in some of my episodes, where the outcome is very well clearly articulated. But the outcome measure that they're using or outcome indicator to measure that outcome
is not a valid measure of the outcome. And I'm going to focus particularly in this episode on their choice of indicators, outcome indicators to measure, quote, the rehabilitation of returning foreign terrorist fighters, looking at that indicator, explaining to you how it is flawed, and more importantly, practicing what I preach, and that is providing an alternative valid measure of rehabilitation, of returning foreign terrorist fighters, so that I can't just stand here and
keep whining and complaining that organizations are using the wrong outcome indicators. I've got to go a step further, put up or shut up, actually provide some real valid measures of rehabilitation of returning foreign terrorist fighters so that they can proceed in that direction. Because that's the whole purpose of my podcast is to get these organizations to improve or for that matter, just even start evaluating their projects in a way where they can actually claim
with valid data. analysis that they actually are achieving the outcomes that they keep claiming on their website that they are achieving but they're not currently at least and that distinguishes them from just a charity because we're not disputing the fact that all these wonderful people around the world need all these wonderful services that we all know are being denied to them because of a variety of reasons that's a separate issue so you could be a charity go ahead, deliver all
the services you want and go home. Just call yourself a charity. That's what a lot of charities do. But when you get into this next level where you're claiming that if you give us money, we're not only going to be a charity, we're going to achieve these expected outcomes, then we get into the whole realm of intellectual dishonesty. So put up or shut up, right? So that's what I'm
going to focus on today. And before I do... As I've always been saying, if you live in the Global South or if you work at an international development organization where hopefully you are required to go to the Global South and monitor and evaluate these projects, then you need some useful tools, not just psychometric tools that I'll be talking about, but you also need three things that I highly recommend in the Global South. Sunglasses.
Proper clothing and proper travel gear because you're going to be in the field interviewing people, collecting data, maybe doing some focus groups, whatever. You need sunglasses. So you want to check out akila .la. That's A -K -I -L -A dot L -A. They also do prescription lenses, not just regular sunglasses, but prescription sunglasses, all made from... 50 % recycled material. You get a 10 % discount if you use the link that
I'll provide in the episode notes. And if you do that and purchase, I get a 15 % commission when you purchase any product using that promo code. And so you have your sunglasses to protect you from the sun in the global south, but you need the right clothes to be comfortable in that hot and humid climate. And so you want to go. for your clothes to hypernaturalstyle .com. That's hypernaturalstyle .com where you can get some
men's tees and button -ups. If you purchase from that link in my episode notes, you get 20 % off any product when you use my link in the episode notes and I get a 20 % commission off. from the total purchase you make when you use my link. And you might be wondering, so what is so cool about these hypernatural products? They have a patent pending hypercooled jade technology that when it's hot, lowers your body temperature
by three to five degrees. They're also breathable, thanks to Supina cotton, and they are sold in premium retailers like Nordstrom. And we're rated the best. men's polo shirts in 2024 in Men's Journal, Forbes, and Esquire. And the last product I've recommended, which is new, that I've discovered, which is good, is Torain, T -O -R -A -I -N .org. That's Torain .org. They sell backpacks, totes,
wallets, and bags. And what's good about them in the global south is they're durable, they're lightweight, they're water resistant, in case you're going during the rainy season perfect for travel and outdoor use if you are in the global south particularly in the field monitoring and evaluating a project all terrain products are crafted from upcycled feed and rice bags sourced globally each bag's interior is lined with recycled plastic bottles handcrafted by
artisans supporting fair trade By purchasing a Touraine bag, customers contribute to global sustainability. You get 15 % off any product if you use my link that is in the episode notes. The link is mydeals .page forward slash 1HJL. When you do that, I get a 15 % commission of the total cart purchase. So just to refresh your
memory. The Global Community Engagement Resilience Fund on their website is responsible, even though it's a fund, they're still responsible for the proper evaluation design of the projects that they fund, or as they say, grant. So at the grant level, this is quote from their website, at the grant level, each project is guided by a theory of change and a results framework with outcome. and output indicators requiring grantees to collect baseline data, report quarterly, and conduct
final assessments. So it's clear that GSERF is responsible for making sure that the design of the evaluation is proper. That means valid outcome indicators. That means proper statistical analysis to support their continued claims that they're achieving their outcomes. On their website, they clearly state four outcomes we've talked about before, but they also have done a new expected outcome that they claim they're achieving, and that is the rehabilitation of returning foreign
terrorist fighters. So we're going to take a closer look at their results from round one and round two funding in the Western Balkans. Report 2020 to 2025. Again, They say it's an abridged version of the original paper, but I can't find the original paper. So to be fair to them, maybe the entire paper includes all of the outcome indicators. But what I've read so far is there's only one outcome indicator from this report on actual measurement of rehabilitation of returning
foreign terrorist fighters. So we're only going to look at that one indicator. So we're going to start with just reading. from page one, paragraph two, or sorry, page two of that report, paragraph two. And I quote, GSURF has worked in the region since 2017. It started by supporting preventing violent extremism work in some foreign terrorist fighter communities of origin. And then later, reintegration and rehabilitation of returnees
in Kosovo. In 2020, with the learnings of its experience in Kosovo, G -Surf approved its regional strategy focused on rehabilitation and reintegration covering Albania, Kosovo, and North Macedonia, later expanding to Bosnia and Herzegovina. So we've got one, two, three, four countries in the report. I see them. Albania, Bosnia and Herzegovina, Kosovo, and North. where they're involved in funding organizations that are delivering, among other things, I presume, psychological services.
Those are your outputs to achieve the expected outcome of, quote, rehabilitation of returning foreign terrorist fighters. So there's a background for you. So if you go to page three of this report, figure three, it says, quote, progress on regional strategy indicators that represent key dimensions of its rehabilitation and reintegration strategy. And this is one of the problems you get into when the people at GSERF who are responsible for performance and impact have got the wrong
university degrees. They've all got degrees, undergraduate, even undergraduate, bachelor's, master's, and PhD in the same international affairs subject area. So you obviously know right away that they're going to be obsessed with national level aggregation and even international combining across countries indicators. But that's not what you want here. What you want to see is for an individual project, which they, being international affairs degree people, are obsessed, will say,
ooh, that's too granular. That's too granular. But that's how you're going to properly evaluate whether these, quote, returning foreign terrorist fighters have actually been, quote, rehabilitated. You can't combine this indicator on reintegration across several grants and then across several countries. Because even if you do, and even assuming your indicator is valid, where are you going to locate which grant is doing well? Which one
isn't? You have to go back. Why do evaluation if you can't find out which grant is screwing up, which one's being effective? So that's a major problem. But let's get into the specifics. First of all, the weakness in the statistical analysis. So they have one indicator on page four that's related to, quote, reintegration
of returning foreign terrorist fighters. foreign terrorist fighters and their families who report participation in social activities with members of receiving community so they start with a baseline percent that actually report participation who actually report participation at 70 percent At the end of the program on page four, they claim it's gone up to 93%. So they're claiming that there is a increase from 70 to 93%. The problem is they're combining it across several grants
in several countries. So we don't know which grant in which country has actually gone up on this indicator and by how much. or gone down. The other problem is they're using an average of this percent. They should use the median because the median, as you've learned from your second year undergraduate stats course, the median is less sensitive to central tendency. Sorry, less sensitive to outlier, large values or small values,
right? It's better to use the median and then maybe compare the median percent at baseline and see if it's gone up to... statistically significant up to 93%. So that's the first problem they have with their statistical analysis. The second problem is they've combined, not only averages is wrong, they've combined all the grants across all the projects within a country and across all the countries that I listed, which is what? How many
countries we got? One, two, three, four. Bosnia and Herzegovina, Kosovo, Albania, North Macedonia. The other problem is I've mentioned in the other episode is this percent. We don't know if it's a sample percent or a population percent. It could be a population percent because when you read in the report, they only talk about on page five, 127 returnees. So they could easily without expense measure all 127, which gets to my more
important point. If there's only 127, instead of... using a biased, poor indicator of the percent who, quote, report participation in social activities and their families, you're going to have a self -reporting bias. It doesn't objectively measure whether or not, quote, they've been reintegrated. It doesn't get to the core. So that's the major problem. The indicator is not a valid measure,
and it's biased also. due to self -reporting bias, in claiming that they're actually been successful at reintegration over time, right? So what do you do instead? What is a valid indicator of reintegration of returning foreign terrorist fighters? Assuming that they're getting, and they talk about it in the report, but I don't want to focus. on these outputs, but I presume they're going to be delivering a variety of different
psychological services. It is good that they've recognized the outcome indicator is the same across all countries, even though there's cross -cultural differences, but they've just chosen the wrong indicator because it's not a valid measure. It's not a standardized, objective, global indicator of measuring rehabilitation.
reintegration of returning foreign terrorist fighters it's not but guess what there are a few valid psychometric tools out there and this is why that background education background in international affairs unfortunately gives them a bias where they're not interested in looking at psychology which is what this whole funding is about psychological services to get these former returning foreign terrorist fighters to rehabilitate right complex so you need to get
a proper psychometric tool that looks at it so you might be wondering what is there out there so this is what i did i just entered into co -pilot the following question what is a valid psychometric tool to measure a reduction in an individual's predicted threat level over time where this individual was a former terrorist
fighter. Up comes from co -pilot, quote, the most widely accepted valid psychometric and risk assessment tool specifically designed to measure changes in an individual's terrorist threat level over time is the, get ready for it, violent extremism risk assessment. V -E -R -A, Vera, version two revised. Now I thought, this is co -pilot. Sometimes you get wrong answers. So then I entered that into just a Google search and up came a website.
www .vera -2r .nl backslash instrument. And lo and behold, it's been developed by the Dutch. But let me just give you a little bit more detail. Quote, unlike standard criminal risk tools, the VERA2R operates on a structured professional judgment framework. This methodology allows psychologists, intelligence professionals, and correctional staff to reevaluate a former fighter at regular intervals to observe whether dynamic risk factors are decreasing and protective factors are increasing.
While several tools exist, two primary instruments are internationally recognized for monitoring a former fighter's rehabilitation and trajectory over time. It is an objective, evidence -based tool optimized for individuals who have been incarcerated or detained for ideologically motivated violence. By the way, if you're listening to this on Spotify, feel free to make comments.
If my critique is flawed in any way, if I'm using the wrong psychometric tools, let everybody know what we should use because the taxpayer is paying for all of this, right? Measuring change. The tool specifically tracks 34 distinct indicators across five domains because many of these factors are dynamic, changeable. A practitioner can run the protocol every six to 12 months to chart an individual's progress or regression. So this
is what GSERF should be using. The beauty of GSERF being a fund, they now have a tool here. All they have to do is find an organization that will be subjected to the training and use of the tool. Then they can just report back to GSERF on the indicator, right? So there's another one, ERG22 +, ERG, Extremism Risk Guidelines, developed in the United Kingdom. The extremism risk guidance 22 plus is used heavily by forensic psychologists and probation officers to evaluate convicted
terrorists. And you can Google that and you'll find there's a publication coming up on researchgate .net. The application over time as a former fighter undergoes rehabilitation programming. The ERG 22 plus maps whether their intent to commit has deflated. even if they still hold a lingering portion of their original ideological engagement. Just to give you a little bit more, psychometric tools in this domain do not spit out a numerical
score like an IQ test. Doing so is considered highly dangerous by security experts due to the low base rate of terrorist events. Instead, they measure threat reduction by looking for specific behavioral changes. Two more. Depending on the specific context of the former fighter, two other frameworks are occasionally paired with the VERA -2R. One, the TRAP -18, Terrorist Radicalization Assessment Protocol, TRAP, a tool consisting of eight proximal warning behaviors and ten distal
characteristics. It is highly useful if security agencies need to determine the imminence of a threat, or if a former fighter is actively planning a relapse into operational activities. You can actually buy the TRAP -18 online. Another psychometric tool, the RAN. That's the European Radicalization Awareness Network. It's developed by the European Radicalization Awareness Network. This tool is optimized for foreign terrorist fighters who
have returned from... conflict zones. Sounds pretty relevant to what GSURF talks about, returning foreign terrorist fighters. They should check this one out. This tool is optimized, blah, blah, blah, to measure their specific reintegration risks. So that even fits with the second part that GSURF is claiming they're achieving, which is reintegration after rehabilitation. Now, there is an indicator on reintegration, but I don't
want to talk about it right now. And there's even a paper that reviews all the psychometric tools. But you can find that out online. So what about buying it? You cannot buy the Vera 2R manual or its psychometric protocols. So what you have to do is it's officially offered by the Netherlands Institute of Forensic Psychiatry and Psychology. And what they will do is they'll provide the certified training. curriculum development and user registries across Europe and works with
licensed global trainers internationally. There's even regions outside of Europe that also are periodically licensed to run authorized cohorts. So if you contact them, you can learn more about the training. So again, GSERV, not being an NGO that delivers the services directly, they can get an organization. in their request for proposals that's willing to do that. And they could just
fund them. And they'll go off and train these people using this valid outcome indicator on the reintegration of returning foreign terrorist fighters. And then all they have to do is come back to GSURF and report on this indicator as part of the evaluation framework that GSURF, that is, they're responsible for developing. So included in the purchase is the course curriculum, the official manual upon successful graduation,
passing the qualification parameters. You're handed the physical VERA 2R professional manual. And then you're also, this practitioner is added to an elite monitored international register, which is great because if you can get local citizens in these countries. Kosovo, et cetera, to become practitioners, you could even argue that's one of the other outcome indicators that GSERF keeps claiming they're achieving, which is vocational
skills training. So they've also achieved that and also additional income, increased incomes. So that's a great example where if they use this psychometric tool, they can achieve two outcomes at once. They can achieve an increase in vocational skills, followed by an increase in subsequent income. And hopefully if they're using that indicator, as I recommend they do, and also using these valid measures of rehabilitation of returning
foreign terrorist fighters. So that gives you an example of how their choice of outcome indicators are not valid, and therefore they can't claim. like they are claiming that they're being successful at rehabilitation when they aren't, for the reasons I provided. They're not using a valid measure. I have provided, what is it, four valid measures, psychometrically, of rehabilitation of returning
foreign terrorist fighters. To be clear here, the indicator that they have provided in this report that I've reviewed, that as invalid, is actually a measure of reintegration because they're asking the self -reporting bias of the returning foreign terrorist fighters to report on their participation. along with their families, which also is self -reporting bias, as to whether they have participated in social activities with members
of the receiving community. That clearly suggests this is an indicator of reintegration, which further stresses the point that in the report that GSURF has produced, there's not a single indicator they have shown in that report that measures validly. rehabilitation. And so the four measures that I've offered, they should be looking at those. That's very important because unless I've missed something, those indicators, there's not a single one that deals with measuring
validly rehabilitation. Now it might be in the
larger report. We don't know, but you'd think they would have it in at least the shortened report that they've posted on their website because they're claiming that they have been successful in rehabilitating these fighters and obviously that's a precursor you first have to rehabilitate them before you reintegrate them right so that even stresses the point even further where are your valid indicators of rehabilitation because they're not there so g -serve has four that i've
suggested in this episode that they should draw on so what i'm going to do now is since g -serve has received money from the government of canada in the past and they are listed on the Global Affairs Canada project browser website. I will be, as usual, sending an email to the Secretary of State for International Development, along with copies of that email. They'll be copied
on it. All the opposition party critics, shout -out critics on international development, recommending that they use this indicator for any organization, including GSURF, that they're funding to deliver services. that the Canadian taxpayer is paying for to achieve this expected outcome of rehabilitation, of returning foreign terrorist fighters. Also, recommending that they follow the lead of GSURF in actually putting the performance measurement frameworks, that's what they're calling Canada,
on the website. of the organization that's received the money from the Government of Canada. And even better, that they put them on the Government of Canada website, which they are currently not doing. So, thank you for listening. Bye for now.