Get my free whitepaper “10 Ways AI Will Transform Talent Acquisition” – Download Now
Transform Talent Acquisition – Find out how my digital course will help you shape the future of talent acquisition in your organization – Click Here
Assessment is the one area of talent acquisition practice where we have proper peer-reviewed science about what is and isn’t effective. Unfortunately, the use of science in assessment has tended to only happen in larger organizations and then not necessarily consistently.
AI has the potential to change everything, making assessment technology faster and more accessible to employers of all sizes. However, there are huge dangers here. Not only does AI bring with it the potential for more bias in the process, things are moving so quickly that it may well be that some tech solutions coming into the market are built without due consideration of the fundamental principles of science and psychology that should be applied to assessment.
So, what steps can TA Leaders take to ensure their technology is fit for purpose and embed a more strategic approach to assessment?
My guest this week is Dr Charles Handler. Charles is the President & Founder of Rocket-Hire, an organization that designs, implements, and validates employee selection solutions. He is also a Futurist, working extensively to research and understand the impact of AI on the future of assessment.
In the interview, we discuss:
• How assessment has evolved over the last five years
• Making the science more accessible and assessments more user-friendly
• The importance of properly evaluating assessment technology.
• The opportunities and threats of AI
• Finding a positive way to respond to the growing use of AI by jobseekers
• Skills-based hiring
• How to think more strategically about assessment
• What does the future look like
Follow this podcast on Apple Podcast.
Matt: Hi, this is Matt. Just before we start the show, I want to tell you about a free white paper that I’ve just published on AI and talent acquisition. We all know that AI is going to dramatically change recruiting, but what will that really look like? For example, imagine a future where AI can predict your company’s future talent needs, build dynamic external and internal talent pools, craft personalized candidate experiences, and intelligently automate recruitment marketing. The new white paper 10 ways AI will transform talent acquisition doesn’t claim to have all the answers, but it does explore the most likely scenarios on how AI will impact recruiting. So, get a head start on planning and influencing the future of your talent acquisition strategy. You can download your copy of the white paper at mattalder.me/transform. That’s mattalder.me/transform.
[Recruiting Future theme]
Matt: Hi there, welcome to Episode 631 of Recruiting Future with me, Matt Alder. Assessment is the one area of talent acquisition practice where we have proper, peer-reviewed science about what is and isn’t effective. Unfortunately, the use of science in assessment has tended to only happen in larger organizations and then not necessarily consistently. AI has the potential to change everything, making assessment technology faster and more accessible to employers of all sizes. However, there are huge dangers here. Not only does AI bring with it the potential for more bias in the process, things are moving so quickly that it may well be that some tech solutions that come into the market are built without due consideration of the fundamental principles of science and psychology that should be applied to assessment.
So, what steps can TA leaders take to ensure their technology is fit for purpose and embed a more strategic approach to assessment? My guest this week is Dr. Charles Handler. Charles is the Founder of Rocket-Hire, an organization that designs, implements, and validates employee selection solutions. He’s also a futurist, working extensively to research and understand the impact of AI on the future of assessment.
Hi Charles, and welcome to the podcast.
Charles: Hey, thanks so much for having me. This is one I listen to a lot and it’s always amazing to have the opportunity to be a guest on podcasts that you listen to.
Matt: Well, the pleasure is all mine, so please could you introduce yourself and tell everyone what you do?
Charles: Yeah. So, I am an organizational psychologist. I’ve been doing that for 30 some years and I’m a specialist in talent assessments from a lot of different angles. From the compliance angle, I build assessments, I field assessments for global enterprise companies. I’m an analyst for the industry so I kind of track the vendors and what’s going on. I’m totally immersed in the AI aspect of things, the ethics, and the efficiency, etc., of all that stuff. I’m the host of the Psych Tech @ Work! podcast on which you’ve been a distinguished guest and just an all-around fan of good hiring and good people practices.
Matt: Excellent stuff. So unsurprisingly, I want to talk about AI in a bit more depth. But before we do that, I suppose by way of sort of context and background, let’s talk about just about how assessment has kind of evolved over the last sort of five years or so. What’s been changing, what’s been going on? What’s the sort of the general direction of travel been?
Charles: Yeah. Well, compared to a lot of other industries and even HR tech, it’s unfortunately hasn’t evolved at the level that other things are. And part of that is because what we use and we know it works, but we have to be so careful. When you’re evaluating people for important, high stakes things like jobs and promotions, you absolutely have to make sure that what you’re using is relevant and fair, etc. I think as well, the long-term providers in the space, they have a lot of business. They don’t have a lot of incentive to change rapidly, nor can they really do that from a technology standpoint. We do see a lot of exciting startups. We see things that people are doing with AI and pretty much everything new now has AI built into it.
The things that I see happening more are definitely a level of user friendliness for candidates. I think in the past you might have been hit with a 200-item assessment with questions that don’t really seem job relevant at all. We can’t get away with that. So, we’re seeing shorter, we’re seeing better UIs, we’re seeing the opportunity to use mostly natural language processing, to listen to what people say or type and be able to pull some elements of predictive ability out of that. I think that the biggest change in the industry is not necessarily coming from the industry itself. It’s coming from providers who are offering the ability to go out and find, source candidates and bring candidates into the funnel using AI and the type of technologies I talked about. They don’t have people like myself at the table a lot of times. So, what they’re saying is we can predict who’s going to be successful in a job, but they’re not measuring things about people in the way that we know how to do it in my world.
Matt: No. That’s interesting. That’s an interesting point as kind of follow up to that. Is the science becoming more accessible as we see more assessment platforms opening up, different price points? Is it more accessible for employers to use sound science in their selection processes?
Charles: Yes, absolutely. So, it’s getting built into some of the platforms, as you mentioned. So, I’ve tracked the market and providers for 20 some years, and every certain amount of years depends on what’s going on. I’ll publish, you know, here’s a list categorized of all the folks that are doing predictive hiring. My database is now up to 400 plus. I just added two today. I added a couple a few days earlier. They’re popping up everywhere, which to me says, “It has value.” What we’re doing has value, which is a good thing. And it is becoming more accessible because of all those various options. And it’s coming down market a little bit. What you have to watch out therefore is from a science standpoint, the more you can really look at a job and study the job and build a set of predictors relative custom to that job, the better you’re going to be. But oftentimes it’s really hard to do that at scale. And to your question of accessibility to science, a lot of times we and I personally have stood there and said, “Hey, we know how to do this right. Give us the data, give us the subjects, let us get in there.” It’s very difficult sometimes, the commitment to really doing it right on the client side is often the biggest limitation.
Matt: Which I think to me and in some ways, I’ve only really started to properly think about this in the last couple of years, to me is kind of crazy. This is the only part of talent acquisition where there is proper peer-reviewed science about how to do something. But it seems that it’s not. It’s kind of ignored and not adopted in the way that it should be.
Charles: Yeah. Well, what you see is if you have people like myself kind of running an assessment program in a global enterprise company, which by the way is happening more and more and more. And I speak from the global enterprise perspective because that’s where I’ve lived and worked for 20, 30 years. But those folks typically have a lot of buy in. They have mature programs or programs that are evolving and the business commits to giving them what they want. What I found as an external person coming in as the voice or representation of what I do, it is often a lot harder and I deal with a lot of high-volume jobs. So, taking an employee off the floor, off the seat, whatever it is getting them, sometimes they’re not allowed to use their own personal time, most of the time to answer because we have to. I’ll back up. The paradigm for making it successful is typically you’re going to study the job. And you’re going to talk to people who do the job and be able to understand, “Okay, what is the job requirements that we want to evaluate.” Otherwise, you’re completely wasting what you’re doing.
So that takes some effort. And to do that right in a compliant way needs to be a nice size engagement. Once you get an assessment in place typically, we do what’s called a criterion validation. We give the assessment to a healthy sample of incumbents and then we do some correlational model building work to see, “Okay, what things predict, what things that we’re excited about predicting that we need. And that takes pulling people off the floor again, often supervisors rating their employees. So, there’s a lot of surveying, there’s a lot of time. It doesn’t have to be a massive commitment, but contact centers, retail, getting people off the floor to participate, it’s a heavy lift sometimes. So, it really has to be emphasized from the very top. And I think that’s what I’m talking about.
Matt: We obviously mentioned AI right at the beginning and you mentioned some of the tools that are coming in that might not necessarily have the proper kind of basis behind them. Is AI being used select people at the moment? And as we move forward, what talent acquisition professionals need to look out for when it comes to the use of AI in this kind of way?
Charles: Yeah, that’s a great question. So, I just mentioned a massive set of friction points, right. Companies are recognizing through their actions that those friction points can be kind of overwhelming. Hiring has a lot of friction anyway. So, if you look at AI, what do we want to do? We want to use it to reduce the friction, maybe to understand what a job requires without doing it hands on. Maybe to understand what a possible employee or a job applicant brings to the table. We can use AI in ways to do that. So, building those matching algorithms, AI can do that. So, to my point earlier, there’s a lot of people who are going all in on that, who may not have our background, who are really empirically driven. That’s one segment of what’s going on. If you move closer into the assessment industry, what you see is that the– and I’m studying this, I’m interviewing people in organizations that do what I do about how their companies use AI. I’m consulting, I’m watching.
So, we’re using it in the industry to build stuff. So, in other words, the risk of saying, “Ah, man, I got to build all these structured interview questions for the job of software engineer.” AI, ChatGPT whatever can do can get you 80% of the way there really fast. So the drudgery, just like in every other area, we’re using AI to– But that’s not facing candidates, that’s not making evaluation. So the next step, where are we doing that? It’s mostly natural language processing. It’s where you’re looking at words or speech and pulling features that are predictive out. And we’re seeing a good deal of that. What I would say too and there’s more, but I don’t want to ramble on. So, what I would say to organizations considering looking at this stuff is you got to evaluate any AI tool the same way you would evaluate any predictive hiring tool. Is it predicting differentially for applicants in one bucket or another? That’s a really important thing. Is it applicant friendly? If it’s completely stealth, are you letting candidates know that you’re doing that?
And I would just say, look at, I built a model called the ETHICS-Hire model, where here’s several things its initials, and each one of these things is going to be something that you need to critically look at when you’re evaluating a vendor. And I think that hasn’t changed. What has changed is the complexity of what you look at. So, if you’re looking at models and, oh, it’s using a large language model or it’s using a deep learning model, it’s pretty hard to unpack those and say, “Oh, this is what’s happening here.” Whereas traditionally in our field, it’s just algebra. The models are algebra. You could take a look at that. You could look at the data that was used to build those, and there’s a lot of clarity there. And so, it’s getting more complicated.
And the last thing I’ll say on that is if your organization has an AI governance model, it needs to extend to everything. So, there’s some general high-level things there that you need to be looking at. You must apply that to looking at a hiring tool. It has to be relevant to the job. You could have the best tool in the world. If you’re not getting in there and understanding what the job requires and you’re missing that calibration, that connection, it’ll never be able to predict what you’re looking for.
[music]
Matt: I suppose on the flip side of this, we’re seeing a big impact now from job seekers and candidates and applicants using AI themselves in the process. What are the implications of that you think?
Charles: Well, the implications are we better get with it and figure out how to deal with it because it’s not going away. So that’s the first thing. There’s some good research that I’ve consumed and that’s pretty available out there that says, job seekers expect to be able to use generative AI on the job, especially younger demographic, graduates, early career. They’re using this stuff every day. People expect to use generative AI on the job whether they’re supposed to or not, especially younger demographics. And there’s been a lot of research that shows that and they don’t want to work for a company that doesn’t allow them to do that. And so, there’s a couple of things, several things that need to happen. We need to understand it’s here to stay and that it’s going to be something that folks do. To preempt that you can either fight it or you can go get with it and say, “How can we understand that?” And turn lemons into lemonade. And that would be what’s the policy, right. Helping job seekers understand what your policy is, potentially even giving them some coaching and tips and things like that about, this is how we would like to see it done. And even more so than just changing some of the methods.
So ChatGPT, this is not even omni 4.0, which is even more powerful if you use this stuff. It can complete a personality test, if you tell it what job, it can complete a cognitive test really well, just like in other disciplines. But there’s types of tests, more interactive, experiential, like simulations and things that make it much more difficult for this to happen. So, you need to start, and the industry needs to start looking to those types of things. There’s, “Hey, do you want to go the old-fashioned way and bring someone in in person and sit there and watch.” And there is a level or kind of a thing I call AI Darwinism. So, one of the funniest stories I like to tell is I was sitting at a conference with people whose job it was to do test security, and multiple people told me they’ve busted people using ChatGPT in a recorded interview, because they could see it in their glasses, they could see the reflection of it. But at the same time, you can use ChatGPT to help you prepare for an interview, to role play and do mock interviews. So that’s valuable.
So, I’ve come up with a concept I call AI-enabled work ethic. And what that means is that’s something, that’s a competency you need to actually evaluate in applicants. Are they able to use this stuff in a creative and positive way on the job? Because that’s something that you want to happen. So, we got to get with it. It’s not going, it’s just like any security thing, it’s going to be hard to keep up. So, let’s look at it a different way.
Matt: No, I 100% agree with you. I think that’s really the only way forward with this and a really positive way forward as well. I suppose, to almost by way of summary of some of the things that you’ve already said, how should TA leaders’ employers be thinking more strategically about these test assessment tools and technologies? What would your advice be to people?
Charles: Well, my advice would be, this probably goes without saying, “Look to where there’s a problem, right” But also look at the bigger picture. I think a lot of times assessment is used tactically to fight fires. Oh, we have horrible turnover in this call center, etc. In the UK, and I’ve had the pleasure of working with a bunch of UK-based companies for graduate recruitment it’s a lot more common to do more of an assessment center or a role play type. So, these are things that I feel like have more strategic value because you’re often looking at the overall competencies and things that, skills that people need to bring to the table. I think it’s really valuable to think about the skills-based hiring paradigm where you’re looking past academic credentials, etc. Assessment is critical for that. If you’re just using inferential methods, some kind of AI technology to try and tell you this person has these skills, for hard skills, that works okay, for soft skills, we’re not really there yet, so don’t rely on that.
Don’t believe the hype I would say, the paradigm that we use works really well. I would also say evaluate your vendors very carefully. I do a lot of vendor evaluation and companies. The first part of our exploration is, “What in the heck are you looking to do here? Who are you? What are the parameters that will make an assessment program successful?” Okay, now let’s go. Look, don’t just say, “Oh, I got. I watched a webinar from this provider, it looks like they’re doing amazing stuff. So and so uses this provider, although that is valuable feedback who your peers are using.” You got to understand before you go shopping, you got to make that shopping list and then you’ve got to be very thorough and critical about it. Take the assessment, ask for the—If they can’t provide you with a technical manual that shows how it was built and the actual science that goes into it with a good sample size, etc., you’ve got to run screaming because the minute you get audited by someone, you’re in trouble. But even more so, that shows a level of responsibility I think that’s really important.
And the last thing I’ll say is, I can’t overemphasize this. I get into this battle of name that tune sometimes in some sense, like, “You need to hire this person in five minutes or less.” And I’ll say, “Well, give me 10 minutes, maybe I can do it in 10 minutes.” You don’t get something for nothing. And the other thing that’s important is people always say, “Well, applicants aren’t going to sit for an assessment.” Well, that may be some of them, but I’ve also seen great research from the candidate experience folks there, the talent board, they have done some great research that show– they interviewed or– excuse me, survey job applicants, thousands of them. And the preponderance is we want a chance to demonstrate our skills. We want a chance to show what we can do. And so, the more job related, another mantra I have, candidates never leave a candidate asking, “Why are they asking me this?” Or wondering, “Excuse me, why are they asking me this?” It should all be very clear.
So, a 20-minute assessment is not bad. There’s good research that shows the inflection point is around 20 or 25 minutes. And in fact, candidates view very short assessments as less credible. So, job related in a good timing. They’re engaging, if you can. I think those are all really, really important things to think about. Don’t limit yourself by saying, “We don’t have enough applicants.” You know, that’s going to drive applicants away. Do you really want to hire somebody that’s not going to give you 20 minutes of their time? I know I wouldn’t. So those are all things to think about. There’s barriers that get thrown up, don’t believe that hype either.
Matt: And so, as a final question to you, where is all this going? What does the future look like? How are things going to develop? What’s going to happen over the next two or three years?
Charles: That’s a great question. It’s the one everybody’s asking, and I don’t claim clairvoyance, but what I can say is I’ve dedicated my whole career to thinking about this. We’re not going to have tests anymore. You’re not going to take a Likert scale personality test in the future, absolutely not. There’s going to be two ways that it goes. Way number one is complete stealth. You’re going to be evaluated and probably eventually pretty accurately through the preponderance of digital exhaust that you have through some other application standards like interviews and materials that you send in. So, there’ll be the stealth aspect and then there’s going to be the hands-on aspect. We’re going to be able to simulate jobs, interactions and work product on jobs at scale much more easily than we ever have before. And think about it, that’s been a longstanding truth. You want to see how someone will do on the job, give them a piece of the job to do for goodness sakes. So that has been difficult to do at scale in a very precise way. As I mentioned earlier, that precision is just really important.
So, I believe that we’re going to see no assessment and we’re going to see highly job-related assessment. Those two can fill different purposes as well maybe the no assessment is at the top of the funnel. We’re already starting that. But again, the inferential stuff is typically not being driven by science. It’s being driven by data science, empiricism, and not a blend of rationality and empiricism and that’s a problem. And the simulation world has been limited. So, that’s the kind of stuff I’m working on. Like, I’m betting my future [laughs] on simulations. Large language models can role play. Large language models when they be text to video is accessible. You’re going to be able to simulate, you and I having a podcast and the dialogue and everything that’s happening but we won’t be there. And so that is going to change the game. That’s going to change the game.
Matt: Absolutely. We live interesting times. Charles, thank you very much for talking to me.
Charles: Yeah. Thanks so much for having me on. It’s a real honor and I appreciate it.
Matt: My thanks to Charles. You can follow this podcast on Apple Podcasts, on Spotify, or via your podcasting app of choice. Please also subscribe to our YouTube channel by going to mattalder.tv. You can search all the past episodes at recruitingfuture.com. On that site, you can also subscribe to our newsletter, Recruiting Future Feast, and get the inside track about everything that’s coming up on the show. Thanks very much for listening. I’ll be back next time and I hope you’ll join me.
[music]
[This is my show]
[Transcript provided by SpeechDocs Podcast Transcription]
