The changing nature of assessment is something we have covered from several different angles on this show before with contributions from both practitioners and consultants. However, to date, we haven’t actually heard the vendor point of view in all of this. So with that in mind, my guest this week is Sam Whiteman, Head of Product Strategy at SHL.
In the interview, we discuss:
- How technology has evolved assessment over time
- Using AI as a capability to drive specific outcomes
- How AI can be used to find hidden gems in the candidate pool
- The evolution of video interviewing
- Giving candidates more value by building an interactive assessment experience
- The importance of people and machines working together to be better
- The problem of algorithmic bias and how to avoid it
- The future of assessment
Subscribe to this podcast in Apple Podcasts
00:00
Matt Alder
Support for this podcast is provided by SHL. From talent acquisition to talent management, SHL has the people insight to help you build a talent strategy that achieves outcomes like increased productivity, internal mobility, engagement and leadership diversity. SHL brings transparent AI technology, decades of trustworthy data science and objectivity to help companies attract, develop and grow the workforce you need to succeed in the digital era. Visit shl.com to learn how you can unlock the full potential of your greatest asset, your people. Hi everyone, this is Matt Alder. Welcome to episode 251 of the Recruiting Future podcast. The changing nature of assessment is something we’ve covered from several different angles on the show before, with contributions from both practitioners and consultants. However, to date, we haven’t actually heard the vendor point of view in all of this.
01:09
Matt Alder
So with that in mind, my guest this week is Sam Whiteman, Head of Product Strategy at shl. In our wide ranging conversation, we talk about the role of AI in assessment, the evolution of video interviewing, and the balance between humans and machines. Enjoy the interview. Hi Sam, and welcome to the podcast.
01:30
Sam Whiteman
Thanks Matt. Thanks for having me on.
01:32
Matt Alder
A pleasure to have you on the show. Could you just introduce yourself and tell everyone what you do?
01:36
Sam Whiteman
Sure. I’m Sam Whiteman. I work at SHL. I lead our talent acquisition solutions. We’re a company headquartered in the uk. I was born in New Zealand, worked in Australia, now living in the San Francisco Bay area and delighted to be on the podcast today to talk about how SHL has evolved and how we work with heads of recruiting and their functions around the world.
01:58
Matt Alder
Let’s sort of pick up on that straight away because SHL will be a sort of a familiar name to lots of people kind of all around the world, but people may have a somewhat outdated view of what you actually do. So tell us a little bit about how the company’s evolved.
02:12
Sam Whiteman
Yeah, so certainly SHL has been around for a long time, Matt. We have a 40 year history and obviously assessments and talent analytics and insights from people data back in back 40 years ago looked very different than it does now. We have had a very strong history in leading the field from a science point of view and today we’re still the largest private employer of industrial and organizational psychologists, most recently. So fast forwarding maybe 38 years. We are now a standalone business operating globally in private ownership. And that assessment and that science is still really important in terms of how we look at people analytics globally. But also there are other things that I’m sure we’ll touch on today in terms of what does that mean for the workforce of today.
03:02
Sam Whiteman
And also, what does that mean for the candidates or the employees that are going through some sort of assessment experience?
03:10
Matt Alder
Absolutely. Before we talk more about assessment, it would be good to talk about digital transformation, because obviously that’s a challenge that lots of employers have at the moment in terms of how they digitally transform their business. And it sounds like that’s a journey that you guys have been on in the business as well. So tell us a little bit about that.
03:28
Sam Whiteman
Absolutely. And I think this is part of how technology has evolved and the use of assessment in particular, and just the. And just the advent of technologies that we can use alongside our science to bring better experiences and then better insights and then better outcomes to our clients. I think digital transformation is one of those things that has been thought of as necessary for technology companies, but it’s often more valuable for companies that don’t historically work in technology or don’t sell technology. I know, and I’m sure, because you’re right in your book, you may have heard the story, but if you were offered to get IPO stock in Google in June 2004, I think it was, or invest in Domino’s Pizza, you would have actually made more money if you invested in Domino’s Pizza.
04:20
Sam Whiteman
And that’s because they were able to digitally transform a pizza company and do much better than a very capable and growing technology stock. So I think the whole thought that digital transformation is just for technology companies, I think has gone out the window. And we know that if it can work for pizza companies, it can work for hospitals or retailers. And we see that in our clients in terms of how they want to acquire different talents to make that shift into a digital company. And as you say, Matt, that’s something that we have gone through as well from an SHL point of view in the last couple of years.
04:59
Matt Alder
Talk us through the technology here. You sort of mentioned technology and assessment. It’s a very sort of big topic. How has it evolved? What technology are you using? What purpose is it serving? Give us the kind of story there.
05:13
Sam Whiteman
Yeah, sure. Well, it’s actually a reasonably long arc, if you think about, and I’m sure we’ll talk about AI later, but if you think about some of the technology that has helped with assessment in the last 10 or 15 years, that was the introduction of things like item response theory and computer testing. And for those who are technically minded, that was a big breakthrough in the IO cycle. Now that’s technically using machine learning. Right. And so it was sort of using machine learning 10 years ago, 12 years ago, but before it was cool. But now on top of that, just the pace of change with all the cloud based technology, new experience, being able to deliver a better experience for our recruiting customers and their candidates has meant that we can do a lot more in a much shorter space of time.
06:00
Sam Whiteman
And so if we think about the progress that we’ve made either organically or inorganically in the last couple of years in terms of making assessments faster, bringing more multimedia to assessments using AI, and in both the creation and the execution of assessments, those are some of the technologies that, yes, they are enabled by moving to the cloud and moving to more modern sort of both front end and back end technologies, but also just the people element that actually gets that out the door. And so we’ve invested heavily in people to be able to build on top of those technologies for new products and services to our client base.
06:39
Matt Alder
Talking a bit more about AI because it’s an interesting topic. First of all, there seems to be a bit of a debate about what it actually means when it comes to talent acquisition. I also think that there’s this sense that people fear machines are making decisions about them and how they’re obsessed. Is that a good thing? I mean, what does AI mean to you and how is it sort of being used in the work that you do?
07:03
Sam Whiteman
I think it’s come a long way. And I think first and foremost we view AI as a capability. None of our clients are coming to us and say that they have an AI problem. Just like 12 years ago, they didn’t come to us and say, we have a machine learning problem. You have to build machine learning. They’re coming to us with a specific outcome that they’re trying to inflect, which is, can I produce a wider range of assessments and really drill down into specific competencies across my organization? Can I use multiple types of assessment and blend them in a way that creates a better experience for our candidates? Can I just make stuff shorter? Can I put other types of assessment in different parts of the phase or funnel or use them in different phases or parts of our organization?
07:48
Sam Whiteman
So I think that’s the important approach that we’ve taken, is that first of all, let’s focus on the outcomes first. And if AI can be a part of that, fantastic. If it doesn’t need to be a part of that, then let’s make sure that we get the outcome right first. The other thing that when we look at that as a capability is were, we acquired a company last Year called Aspiring Minds who have a really heavy Data Science and AI team and have a good 8 to 10 year history again in the production of AI capability rather than the production of AI products per se. And so we can dive into that a little bit more, I am sure.
08:26
Sam Whiteman
But it’s not AI and assessment is not just about using AI to assess, it’s about the whole experience, be it the candidate experience or the recruiter experience, or in fact the fidelity or relevance of the actual assessment itself.
08:40
Matt Alder
Could you give us some examples of that working in practice?
08:44
Sam Whiteman
Yeah, sure. So I think one of the best examples, I think, and it falls in the, dare I say, the unsexy side of AI assessment. But one of the things that we’ve been able to do with the Aspiring Minds team is to figure out with coding simulation products. So coding simulations that are taken from, for software developers and technical talent. One of the problems with doing coding simulations in a short amount of time is that it’s a short amount of time. And to get through use cases sometimes what were finding is that maybe 25 to 30% of candidates that would go through the process, their code wouldn’t compile in some or all of the use cases that we put in front of them. Now we tend to think that if a software developer can’t compile code that is a bad thing.
09:31
Sam Whiteman
And I think generally that might be true. But also we know that if we’re only assessing them in a 30 or 60 minute window, the fact that they didn’t have this bracket in the right place or pass a particular use case doesn’t necessarily mean they’re a bad developer. Now so that was the problem again, that was the outcome that companies were having was that 25 or 30% of their candidates weren’t passing the simulation. But what were able to do, and again using people and studying volume, was to make sure that we could artificially grade that code despite those smaller mistakes. So we could simulate what kind of score they would have got if they weren’t making these type 1 and type 2 errors in the process.
10:12
Sam Whiteman
We can flag that as hey look, this is scored by AI knowing that it didn’t actually compile in the system. But that means that we’re able to access a much bigger pool of candidates, again, 25, 30% in some instances, so that we’re actually getting the score for them and we can do something with it. And so what our customers have been telling us with that is that they’re able to find those hidden gems where they’re like look, this person ended up really great. But one of their test cases didn’t compile in the coding simulation they took, which only took them 45 minutes or we only gave them 45 minutes to complete. So I think the using AI to grade code that doesn’t compile is one of those great things that has enabled an outcome which is more candidates.
10:57
Sam Whiteman
The other ones I would say is using advanced proctoring techniques. And so where data privacy and candidate disclosures allow, we can tell if two people are looking at the screen while they’re taking an assessment that would generally be bad. And then the other one would be the use of AI to score language assessments. And so how do we get some really quality measures in terms of language which is over and above all the stuff that we would do which is traditional AI and assessment, I guess, if that is such a thing yet which is how we use natural language processing and largely a video interview context to write candidates and score candidates on their fit and readiness for a role.
11:45
Matt Alder
I was going to ask you a question about people and machines, but I’ll ask you that in a second. Just want to delve deeper into video interviewing. How are you seeing that developing and where do you think it might be moving to next?
11:59
Sam Whiteman
Yeah, so I think and again like trying to take an outcomes based approach first and foremost, which is like what are we trying to do here? And is video the right, the right part of that outcome? I think we think that there’s an opportunity for video and assessment to be a, you know, to be a one part plus one equals three type equation and there’s a few opportunities for that. And again it’s about working backwards from what the customer needed for a specific role. Because sometimes it might not be 100% video interview and sometimes it might not be 100% psychometric assessment, but those things can definitely better together.
12:39
Sam Whiteman
And so one of the ways in which we can use that is firstly making sure that we’re using the techniques available, which is with this being able to use AI to score and then being able to do that at scale I think is really important so that we’re not creating these sort of one off lengthy black box type algorithms for specific roles that don’t necessarily flex to other roles. So being able to broaden that out across the candidate pool or across multiple job recs is going to be really important. And then secondly using it to make sure that you can almost communicate to candidates in the flow so they can see that they’re making progress. And so what we can do is, as an example, give candidates a short force of volume roles.
13:21
Sam Whiteman
We can give them a short sort of situational judgment, testy or short psychometric assessment, and then if they get over, say the 40th percentile or 50th percentile, we can then quickly hurdle them into the next step, which might be a video, and you to say, hey, you are progressing through the process and this is the next step, which does give them that energy to put a bit more effort into that process. And then it does also let them see more of that experience to say, well, tell me more about the job then. And so they can receive a lot more value out of the assessment experience as well, because they can then meet the hiring manager virtually or they can find out about the culture or what their job is as well.
14:05
Sam Whiteman
And so there’s going to be a real mix of both give and take in those assessment experiences. And so that’s where video interviews can play a really strong role in communicating to the candidate and progressing them through, but still providing a really quality assessment experiment as well.
14:22
Matt Alder
Fantastic. You’ve talked about people and you’ve talked about AI and machines and the way that they can work together. What’s your take on what is it that people can do that machines can’t? And what can machines do that people can’t?
14:35
Sam Whiteman
Yeah, that’s a great question. I mean, we will see how this plays out. Right? And so I think the easiest thing to say is that people and machines can make each other better. I think we should all start there, which is to say people have been making poor decisions without machines for a long time or suboptimal decisions for a long time, and machines have been making suboptimal decisions for a short amount of time, and people can make them better as well. Again, this is where sort of the people, the humans in the loop and the people making sure that firstly, the ethics and compliance of using data is there. And so you just need people to do that.
15:20
Sam Whiteman
If your algorithm is reliant on a particular job site, and that job site has overwhelming numbers of applicants from a specific gender, then that’s probably only something that a human is going to know when they look at it. Right. And so making sure that those humans are in the loop so that they’re making those appropriate adjustments and understanding the full context of how we’re using the data is going to be really important going forward. So I don’t think there’s any sort of specific evidence, even in 2020, that there’s a move towards automation of jobs for recruiters.
15:58
Sam Whiteman
At least I know recruiters from a role point of view are managing to be much more strategic in terms of coaching hiring managers in terms of what they want out of a role and then coaching candidates to make sure that they’re making a win decision for both their careers and the company as well. And so there’s a lot of human influence both with and without the machine to last us a while yet.
16:22
Matt Alder
I think you’ve given the answer to this already in your last answer, but you mentioned earlier the whole idea of creating these sort of black box AI things that make decision and no one understands how the decision has been made. And that’s something that we sort of discussed on the show a few times in relation to transparency and bias. What’s your take on the potential for bias in sort of automated or technology driven assessments?
16:52
Sam Whiteman
Yeah, so I think there is overwhelming evidence that sort of left unchecked algorithms can be biased. Right. And you know, humans aren’t coming from a place of perfection in terms of that as well. But we, but we have that ability to make sure that we are looking at how we’re building assessments following appropriate guidelines and methodologies and then being transparent about what it is that we’re actually measuring. I think probably, I think most practitioners that are in this space are doing a good job at covering the ethics and the back testing and all of that kind of stuff, making sure our sample sizes are large enough. I say most, not all. I think more importantly though, we’ve got to make sure that we just started those basics of like, are we measuring the right thing?
17:43
Sam Whiteman
One thing I’m probably more concerned about from an assessment sort of efficacy point of view is that a number of the AI algorithms that, that we, that are creeping in, I guess they’re all measuring some other measure which might measure something else. And so it’s like, you know, we can, this algorithm predicts this other score which might predict that. And all of a sudden we’re creating these really weak signals on top of weak signals that end up not actually measuring the thing that we want to measure. And I think this is the, this is the real concern is that we’re trying to measure something specific. It might be conscientiousness or something like that. And so we’re trying to build an algorithm that predicts how they might have scored on a conscientiousness test. And it’s like, well, that’s not the same thing as measuring consciousness.
18:34
Sam Whiteman
So you don’t want to sort of degrade the signal of what the algorithm is measuring and say, look, this is really effective at measuring the effect of the potential score of another test, which might actually be based on another test. And all of a sudden it’s really watered down. So I think more than anything, one thing that we’ve been able to focus on is make sure that we’re measuring the thing. And I literally say that internally. I’m like, what is the thing? Because we want to make sure that we’re measuring it. And then when we produce the results that are measuring that we’re able to say, this is shown to be, this is. These are behaviors that have been displayed and confirmed by subject matter experts that are prevalent.
19:12
Sam Whiteman
And so making sure that we’re measuring the right thing is probably just as important as all of the safeguards, protocols, data sizes, back testing that might happen with AI and algorithms, with an assessment.
19:24
Matt Alder
Anyway, final topic, just digging a bit more into the candidate experience and what that might look like in the future. So where is all this going at? What do you think things are going to look like in sort of two to three years time?
19:40
Sam Whiteman
Yeah, that’s a really good question. I think where we’ve, I think where we’ve come from, and especially I’m talking sort of assessments here, where we’ve come from as an environment of, you know, can it be shorter and can we just make candidates like it more, which are worthy objectives and some of the stuff that we’ve been able to do with the new technologies we’re working on. For instance, is our opq, which is our flagship personality product that’s now sort of 40 to 50% faster for candidates. So those are all solid milestones, I think, going beyond the whole make it shorter and make them happier type themes.
20:17
Sam Whiteman
I think what our clients are asking for us now is around how can we bring the same level of rigor to measuring and adapting the candidate experience, knowing that every touch point is important, how can we bring the same rigor to that experience as we bring to science, Right. In terms of the measurement. And so that’s opened up a lot of opportunities and some great discussions with our clients to say, look, if were going to treat this touch point as a really important one, then here’s what we do in terms of making it more interactive. And so previously were like, if we can stuff a whole bunch of corporate information down candidates throats, they will receive it and it’s like, well actually they need to interact with it, right. And then their time needs to be valuable for them as well. Right.
21:04
Sam Whiteman
So it’s not, it might be short, you know, it might be a 10 minute assessment, it might be a 25 assessment, it might be a 30 minute assessment. Either way, it needs to be valuable for whatever that time is. And so the role that feedback can play as an example of that needs to make the candidate feel like it was worthy of their time. Even if they do three of these things every day. You got to try and make sure that you’re providing value no matter what doing. And assessment companies are in a great spot for that because we know a lot about these candidates right at the top of that funnel because we’ve got that people data about them. And so providing feedback there is going to be really important and that interactivity.
21:40
Sam Whiteman
And hopefully when we put those things together, we can really help build that candidate experience right across the pipeline. And then we can do things like a B test for instance, to say, hey, look, you know, when you throw up four videos, your dropout rate goes like this. But when we did an a B test and they only looked at two videos available, then your dropout rate was less or your MPS went up. So trying to treat that experience with as though it’s a science in and of itself is probably the thing that we’re hearing a little bit from clients now. And we’re certainly making investments at SHL to make that into a reality. So I think that’s where I think the AI stuff is interesting. I think we’re in a great spot. But again, it’s not about the technology, it’s about the outcomes.
22:24
Sam Whiteman
And our companies are wanting to make sure that they’re hiring the best available talent and that talent is making a win decision for their career and the company’s role as well.
22:38
Matt Alder
Sam, thank you very much for talking to me.
22:40
Sam Whiteman
Matt, it’s been a pleasure.
22:42
Matt Alder
My thanks to Sam Whiteman. You can subscribe to this podcast in Apple Podcasts or via your podcasting app of choice. Please also follow us on Instagram. You can find the show by searching for Recruiting Future. If you’re a Spotify or Pandora user, you can also find the show there. You can find all the past episodes at www.recruitingfuture.com on that site. You can also subscribe to the mailing list and find out more about Working with me. Thanks very much for listening. I’ll be back next time and I hope you’ll join me.
