Explaining the CritiCall Personality Test with Dr. Clinton Kelly

Mike, do you want to say anything about CritiCall before we jump in?

And I would say the dispatch environment in terms of our conversation has more than means more than 9 1 1 it's police, fire, it's EMS, it's 9 1 1 utility. Ambulance, there's all sorts of environments where the job that's being performed is very much in a stressful situation. You're under duress and it's a CritiCall incident, meaning that the effects of making good decisions can be very positive and the effects of making poor decisions or performing poorly can have incredibly negative effects.

Implications, including even death. So it's a very important environment to be working in. CritiCall is a product that is a product of Biddle Consulting Group. It runs within our TestGenius suite. And a few years back, we had the great opportunity to work with Clinton and his firm to create, ground up a. Dispatcher specific personality test. And that's what we're really going to be meeting today to talk about.

So personality testing and CritiCall. Why does it matter? Why is it important?

And so both of those pieces of the puzzle are important parts when it comes to making good hiring choices. And so that's why it matters. And these are like Mike said, these are stressful jobs, typically working longer shifts, dealing with situations that aren't always pleasant, and you need to make sure you have the right fit for this job. And so that's really important.

And what these folks will tell me very often, and I never disagree with them is that, I've been in this job so long, I can sit down with an applicant. I can talk to them for five minutes and I can know whether or not they're going to be successful on the job. And they don't say that to me in a way where they're arguing with me saying they don't need to do testing. Not at all.

What we all understand about that particular situation is that, yeah, they probably have a really great feel about who would be successful and who wouldn't be successful. The problem is that particular process doesn't have a paper trail. That explains why somebody made an affirmative decision or why they made a negative decision, and it doesn't leave the applicant feeling very good about the process.

I remember in years past, I applied for jobs where I thought I was super highly appropriate for it. There was a. bogus selection process. I didn't get hired and I felt really bad about that. So one of the things that we want to do on the testing side for CritiCall and for this, these dispatcher positions is we want to provide a. process that's holistic and really one that does what these center directors know that they could do, but creates a paper trail at the same time.

And Jenny, you asked Clinton about why, you want to do the personality testing in this space. And one of the things that we look at in testing and selection is you look at Job performance as like a pie with different pieces in there. And so you can have tests that measure audio data entry and map reading and multitasking and those kinds of things. And each of those pieces of pie are going to represent different aspects of job performance.

And so the goal in a testing process is to try to get as many pieces of that Pie filled up with CritiCall performance dimensions. And some of those are going to be hard skills. Some of them are going to be, soft skills, situational judgment. And many of those things are going to be personality, particularly in this realm, because it's such a difficult job is you want to know as much as you can know from a job related perspective about the personality of this person.

Clinton said, specifically to be able to find a good job. a good fit. And so that's really what our goal is with this particular product right here.

And so there, we have actual data that says, Hey, it doesn't get, and there's no guarantee cause as Mike said, there's a certain amount of, we can't predict up with a hundred percent accuracy, but we show through our study that higher scores on the test, they're more likely to be rated as higher performance by their supervisors.

With a heinous job, it's important that people can adapt and on the fly, they get calls and have to figure out how to handle things. Another one, one of the things we tend to measure is composure. You have to be able to remain calm when you're on a call. Another thing is resilience and not being able to let a certain, be affected one call to the next. You have to be able to let that one go and move on following procedure and policy.

That's also another thing you, while it's important to have the adaptable, you also need individuals in these types of roles who are not going to fly by the seat of their pants, that they know what the policy and procedure is in certain situations. And then one of the other ones is confidences. You have to also be able to take some, I say, be a little bit assertive and be confident on the calls.

When you're in these situations and then lastly multitasking a preference for multitasking and that's pretty apparent for anyone who's been in a dispatching agency. If you've walked into a room, you just see the multiple screens, the phones and things that are going on. There's lots of things done at the same time. So just individuals who have a preference or have a preference for multitasking behaviors is another thing we were attempting to measure.

It's not that you want people who are rated a hundred out of a hundred on all of these particular areas, rather you want them to fall within a certain range. How do you determine that?

So in the, what you can see on the screen here, we created a scale where supervisors rated the individuals who took this test. They rated their performance on these different areas of being adaptable, flexible, ambitious, motivated. And we had them rate those employees. So we had people throughout the U. S. and Canada who are current dispatchers take this test. We had their supervisors rate them on their job performance. And then we looked at, we then evaluated the relationship.

Are there patterns in the data that we're seeing that people who are higher or lower on these questions or on these scales that are more likely to be successful in the job as rated by their supervisors? So their supervisors say they're more successful and what patterns on these different test questions result in better dispatchers. And sometimes you can get lucky. What you can do is you can get lucky.

Like I like it is you can throw spaghetti against the wall and maybe it makes some magical image one time, right? But can you do it again? Can you replicate it? And so what we did here to avoid making sure that we didn't just capitalize on chance, we didn't just get lucky with these questions. Like we threw all this stuff in the black box. And while magically people who preferred to eat, sushi versus pizza are better dispatchers.

We might, you might find that pattern in data once, but we did here, as you see on this slide here, see sample one and sample two, we collected data From all these individuals. And we randomly split the sample in half. We randomly said, okay, you're sample one, you're sample two. We did a random split in sample one. We said, okay, these questions predicted performance. And then we use that, we said, do they also predict performance in sample two?

And that cross validation, that helps us to make sure that we didn't just get lucky the first time. And you can see these correlations here. This is, not to get too deep into the data here, but these correlations, when there's stars next to them, these means they are significant predictors of performance. They are predicting job performance in both sample one and sample two.

And so we're able to show that we're consistently predicting performance and didn't just get lucky in the way of the, some unique, just saying that we just capitalize on chance.

And so what we have in the way we set up this test, they are what we call paired like a forced choice personality test and it's where we have two different statements and we say something like i'm more likely to And on the left hand side, I might say follow the rules and the right hand side is change the rules, And so we have that don't you? Yes, we do have an image of that You That is on right here. Right here is an example.

This is a silly example here, but let's just say I am more likely to eat a hamburger or eat a salad. And so you have the ones on the left or on the right, and people indicate where they fall in this continuum of their preference. We set up the questions, the ones on the actual test to be, so they've look more job related. For example, something like this, I'm more likely to, on the left hand side, make quick decisive decisions or make long thought out decisions.

Inherently, neither one of those is bad. Someone who makes long thought versus quick decisive. Sometimes there could be benefit to one or these the other, depending on the job. But in this particular job. We actually, you need to, you are typically respond, responding quickly. You need to make quick, decisive decisions. And so we were looking, are the people who make quick, decisive decisions, are those who are more likely to be successful in the job as rated by their supervisors?

And so those are the types of things we're looking at with these questions. We try to write them in ways that are like Mike has said, more related to the job.

And but of where we had good data, a little over 300 individuals. And you can see, we randomly split that sample in half. In each one, we get about 150 in those random splits of the sample.

But then we also had some phone calls with a number of different managers of dispatching agencies throughout the country, where we asked them to confirm just again, just as a second, confirm of what we were seeing in the job analysis results, what are most predictive of success of dispatchers when it comes to soft skills or the personality kind of different things, different constructs. And we use that to drive the development of the content.

And one of the things I like to say, we custom developed this, like Mike had said. But whenever you make a test, not just like on any team, on a sports team, not all the players are all stars. We developed these questions, not all of the items worked. So when we originally developed the test, we had over a hundred questions that were where we collected data. Some of those questions did not predict performance.

It didn't matter which side they picked, which side of the statement they were picking. He said, Hey, people on either side were equally as effective on the job according to their supervisors. So those are questions that got tossed. They didn't make the final cut because we said it didn't really matter which side you were on. These individuals, they didn't seem to predict. These items didn't predict performance.

And so that's an important thing when you do this development is to evaluate your item and see which ones are working and which ones aren't. And those got tossed. The ones that didn't work, we said, Hey, it didn't work. We thought we might, but it didn't.

And one of the things that I really love about the report is that Clinton, you and Jason spent a lot of time looking at this overall performance and then certain subscales that had validity coefficients. But then to be able to ask probing questions.

So we have a saying in our environment or work environment here, not everybody is good at everything, but it's important to understand where your weaknesses lie, maybe more important to not think that your weaknesses are strengths, but also to be able to remediate those situations, to be able to understand this isn't necessarily a strength.

So can you maybe walk us through a sample score report and show us, how is it that a person who's a hiring manager might use this report, how they might interpret it and how they might deal with some of these aspects of the scoring and maybe some followup questions in order to get the best utility out of this report.

You'll see at the top right there, I got a highly recommended score. We also give the total time that the person took on the test. We give them a 30 minute time limit. As you can see, I've completed this a number of times, but most people, it's probably 10 minutes or less that are completing this test. It doesn't take a long time to complete. We give them plenty of time because it's not meant to be a speeded test. In other words, we're not trying to say, Hey, finish this. You better hurry up.

If you don't go fast, you're not going to finish. We have some background here about the report and what we want to do is we mentioned on this background is we want you like to really focus on the driving factor really should be the overall score. This overall recommendation, this highly recommended, recommended, or somewhat recommended. And the reason for that is because that's considering the most information that's considering all of the test items that create the overall test.

And what we found from our test, for example, those who are highly recommended, they have a 77 percent shot of being successful on the job. And we, what we mean by being successful on the job is they were rated as above average job performers by their supervisors. So 77 percent of the people who have a highly recommended score, in our data collection, they were above average in their job performance ratings.

If you drop down to recommended, 53 percent of those in the recommended were above average in their job performance. And if you go down to somewhat recommended, only 31 percent of those individuals were above average in their job performance ratings.

So as you can see from those numbers, again, no test is perfect, but you can see there, there is value as you go further down in those scores, you're are more likely to bring someone on who is not going to be very as good in their job performance and that the data bears that out.

They need to be able to, read maps and understand, coordinates and such. So these things are all very important and they're deal killers, right? If you possess these skills and abilities. You have the ability to move forward. If you don't possess these skills and abilities, then you really don't have any real reason to be continuing in the process because we're, the centers are expecting people to come into this environment day one.

So if you take that, you say, X percentage of the applicants have the skills and abilities necessary for success on the job. Now you take them and you run them just those people, through the personality test.

So technical tests of, typing auditory information, entering things in that gives us that predicts differently than a personality test. And that's just the term we call it incremental variance.

And so you're adding unique pieces of information that give you basically greater predictive power.

And so we want to let you know Hey, these individuals, this was, this is based on data from actual people working in these types of roles that we aren't just making this up and saying, Hey, we think these people are going to be more successful. This is what the data has shown us.

Now, what we say here now, we introduce these subscales and we have these subscales and they can provide unique information that then could be useful for prodding in say, a selection interview. And so we have one of the subscores that we have is dependability. And so we give you some information on high scores, what that means for higher scores on dependability and what that may mean for a lower score on dependability.

And then we give you some potential interview questions that you may want to follow up with in the interview process.

Thanks. And like Mike had said, just because someone scores lower on one of these subscales, let's say I scored really you can see on this one, I'm like confident assertive, I'm right in the middle, on that 58 on the confident assertive subscale out of 100. And I've seen this before where someone says, Ooh, Clinton's only a 58 on the confident assertive subscale. I don't know. We, I don't think we should hire him.

What we need to go back to is again, this overall recommendation should be the driving decision. Now, what you can do with this is because maybe I'm moderate on this. You could say, Hey, maybe you want to ask some confident sort of follow up questions that we provide here on this page is in as part of the interview questions. Hey Clinton, tell me about this, discuss this, tell me about a time.

And follow up on those interview questions to see if there's maybe any sort of flags there or training needs or say, is this going to be a problem if we bring him in or follow up there?

And you realize that they talk really softly, there's this, you want to jump out and you want to say listen, in this role, you need to be able to speak with authority. It isn't necessarily problematic that this person speaks really softly. What's what matters is, are they aware that their natural tendency is to speak softly, but they have the inherent ability to ramp it up when the time comes. And I think that's the same with some of these right here and some of these follow up questions.

And just because you have a strength also mean you have to be aware of your strengths, because strengths overused can become weaknesses. And just knowing these things, so you can also probe on some of these where they're high scores, because you say, hey, really good, he's really assertive, but you might want to say, hey, are there times maybe where he's overbearing?

Of these 100 let's focus on maybe then the overall recommendation and you drill down on the subscales in some great interview Yeah,

There's not a big difference between somebody scoring 96 or 97 on a map reading test. It's not enough to be able to make some sort of distinguishing decision between, person one and person two, really, these are the kinds of things that you want to use to determine of these 20, which are the people that we're going to move forward to, and whatever our next part of our recruitment is, even if it's hiring.

And so I could really, we've already talked about this, but I could just really see the tendency to, to think that one is more important than the other.

Now, this test has been criterion validated. And why don't you talk a little bit -when something is criteria and validated that a lot of people call that the gold standard of validation, because it literally predicts some sort of job performance that's correlated with the test scores. But one of the things that's really advantageous about that is that the Uniform Guidelines does allow this criterion validation of which shoot you guys wrote a 280 page validation report.

So our clients are able to adopt that validation report as their own, which will give them all of the defensibility aspects that come with that. They're able to adopt that to their own environment through a very simple process that actually takes only a couple of minutes for them to go and transport that over. Can you talk about that a little bit? I think that's very useful to our audience.

All you have to do under the Uniform Guidelines on employee selection procedures in section 7b There we have a simple process to say is your dispatching job substantially similar in terms of kind of the major work behaviors to where we did the original validation study. And we have a document for you where we have listed out the work behaviors, and you can see here, major work behaviors, and we have it both for the technical skills component of the CritiCall test.

So like more of the map reading, audio comprehension, and also for the personality component, and you need just, all you have to do is have your employees, your subject matter experts, or your current dispatchers or supervisors of the position, indicate how similar those are to, to the job at your organization. And if there's substantial similarity, and this will auto calculate for you, we'll say, yes, you have enough here to be able to transport the validity.

And so you don't have to do your own kind of criterion validation study with hundreds of people that you don't even have. And we've done that for you.

And it looks to me like basically you hit each organization's going to have a series of subject matter experts, and they're just going to agree or disagree that these major work behaviors are similar to those that are required in their environment and from what I've seen in using this transportability tool over many years with people, the universality of the dispatch environment is such that everybody goes through this. And yeah, of course, we have to be able to do this.

And of course, we have to do this. It's a really high likelihood that the criterion validation that was established is going to translate or transport over to any other given organization, even if it's a very small rural organization, or if it's a very large urban organization.

And there's a lot of differences, but this dispatcher tends to be, like I said, tends to be pretty similar, and so that's a good thing is that we usually find people are saying, yeah, this is accurate and we do this and we can transport that validity.

The bottom that's red and it turns green. If it's right there, the green signifies that the work behavior behaviors are considered to be transportable. Thanks for slicing and dicing this, Jenny. I really appreciate that. But it's as easy as that. Number one, number two, number three, number four, number five. And then when they get to the end, it turns green and you can print a copy of that.

You can actually take our validation report that Clinton and Jason authored and you can attach your transparency portability report, easy for me to say, to that validation report, and you can then own that validation report for this position for your organization. And I would like to say that if someone was to call you up and do this boots on the ground from end to end at their particular center.

If they were going to, if you were going to show up, you're going to do a job analysis, you were going to do a test development. You were going to do criterion validation. If you're going to do all this stuff on site. That would be a really expensive process, wouldn't it? Yeah.

When you go through this validation process for this kind of test or any other kind of test the way I describe it is that it's like a coin with two sides. And on one side is the utility and the utility speaks to how well does this selection instrument work to bring in the people that are going to be successful on the job in some way or shape or they're not going to steal from you or they're not going to quit immediately. They're not going to be disruptive or whatever your criteria is.

Then on the other side of that same coin opposite the utility side is defensibility. And defensibility is speaking to what in a worst case scenario, if your selection process gets questioned by some human being with an attorney or some legal authority how well are you going to fare in that particular instance? So can you talk a little bit about all the work that we've done and how that benefits each organization in terms of the utility and defensibility aspects.

So if someone does challenge your hiring process or says, Hey, you didn't hire me because I am a white male. Or you didn't hire me because I don't know, some group I'm too old. You didn't hire, we want to be able to defend it and say, no, we didn't hire you because, you didn't have the right technical skills. You couldn't do these things. You weren't, we showed through our validation efforts we know that lower scores on this test are less likely to succeed.

And so you were screened out because of that and not because of your age and not because of some other issue. And that's what we provide. And that's what they're the uniform. We didn't. I haven't gone into this too much, but the Uniform Guidelines on Employee Selection Procedures provide a playbook. And when it comes to personality tests and criterion validation, Section 15B of the Uniform Guidelines on Employee Selection Procedures provides like a list.

A list of just of things that you need to cover when doing a validation study or that a test creator vendor should cover if they're developing a test Yeah in that report and the report that we have we actually at the very beginning of the report Have a list of all the requirements of Section 15B of the Uniform Guidelines, and we indicate which page or pages of the report cover those sections of Uniform Guidelines to show that we're addressing all of the requirements that we're

doing our homework and you're covered. And so if someone does challenge the process, this is a nice report that then can be provided to lawyers and people who are going to get in those rooms and fight those battles to say, Hey, we've done our homework here and we're making decisions based on good information and not because we just don't like people from a certain protected group status.

And I never, I never argue with these folks because I know that they're right about that, that they have that uncanny ability to be able to sort out who's going to be successful and who isn't going to be successful. We would never recommend that they use that as a strategy because it's fraught with pitfalls, right? The first one is that you don't have a paper trail that shows why you selected one person and you didn't select another person and that in itself can be quite problematic.

And so really when we set out to create CritiCall, what we did without even really knowing is that we created it to be proxy for that center director who's able to spend time with an applicant and in a similar way, know whether or not they're going to be successful on the job, but to do so in a way where it's completely documented step by step. Here's the reasons why they should move forward. Here's the reasons why they shouldn't move forward.

But furthermore to that, what we find, and I would say particularly in this social media environment or day and age, is that it's so important for organizations who are hiring to be good citizens, good stewards in their realm. And so what this critiCall test and even your personality test that's in the CritiCall test does and does so well is it affords the applicant an opportunity to find out not only find out for the organization they're applying for, but find out for themselves.

Am I suited for this job and to do so in a way that is really satisfactory. In fact, what we see with our testing is that if you're like me and you can't multitask, you hate the test so much that you pretty much get up and walk away, don't even finish it. And what you do is you walk away saying, Oh my gosh, I dodged a bullet because I would hate that job. I'm so glad I didn't, get that job so I could work Christmas and weekends and graveyards and all that stuff that I really also don't like.

So these kinds of well constructed tests really serve both of those masters in such a way that it leaves the organization feeling very good about the process that they have in place, as well, it leaves the applicant feeling whether or not they got hired that they had a fair shake and even maybe more than some other jobs. If you're not suited for this environment, you really have no disagreement with that. So do you have any thoughts about that? Does that resonate with you at all?

My, my other thought was, it was like you talked about is we're providing what I call a standardized, a nice standardized process for organizations to make hiring decisions. And while it's not often sexy, people aren't often saying, Ooh, look at that standardization you have in your process and, really admiring it.

That standardization is actually what helps and saves a lot of organizations and actually what can build in some of that predictive power, because oftentimes one of the problems, like you said, Mike, with those individuals who've been in the job for years and say they can recognize who's going to be the best performer, is sometimes problems occur when there's a lack of standardization.

If they didn't use tools like this, they all of a sudden they're free flow and they go in one direction with one applicant, they don't go in that direction with another job applicant, that lack of standardization can then where problems can be introduced into the process and a candidate may complain and say, I didn't get a chance to show what I could do because they asked all this other candidate these questions that they didn't even ask me, okay. We're having a standardized process.

We're giving all the candidates a fair playing ground to show their skills and their abilities and they're fit for this job in a standardized process. There's a lot of value in that standardization.

Do you Jenny, did you have anything else that you wanted to talk about before we end?

So Clinton, as we wrap up, why don't you just briefly tell a couple of sentences about ioPredict, who you are and how they can reach you.

But we also work with organizations when sometimes they've adopted a test from another vendor where it's created, but they say, Hey, we don't know. Did they really put this together? And they reach out to us to do validation studies or to be that other source to, to review and to implement tests that maybe other people have created. So we do that as well.

And thank you for our listeners. Thank you for our viewers.

And now that doesn't mean we'll be paying your taxes for you, but we certainly think of you in that regard. And we hold you guys in high esteem. So it's really great to have you here hanging out with us.