Back in September 2018, I interviewed industry consultant Bas Van De Haterd about his research work exploring emerging assessment technologies. The assessment market is moving exceptionally quickly, and I wanted to invite Bas back, a year later, to update us on what has changed.
In the interview, we discuss:
- The problems with traditional methods of assessment
- Making sense of evolving technologies
- The issue with black box algorithms
- Are micro-expressions useful?
- The power of game-based assessments
- Using technology to broaden pools of talent
Bas also shares several case studies from employers achieving considerable success with new assessment technology and gives us his view on the future.
Subscribe to this podcast in Apple Podcasts
Transcript:
Matt Alder [00:00:00]:
Support for this podcast comes from Avature ats, an applicant tracking system that redefines user experience for candidates, recruiters and hiring managers. Just listen to one of the many ways in which L’Oreal USA has improved their hiring process with Avature, as told by Edward Dias, Director of Recruitment, Intelligence and Innovation.
Edward Dias [00:00:25]:
Since we’ve been using Avature ATS globally, we have been able to massively improve our communication rate with candidates during and following their application. Before, over a million people worldwide would never get contacted, but with the smart optimization and flexible processes, we’ve been able to change that and that’s been a huge achievement.
Matt Alder [00:00:48]:
Visit avature.net that’s a V A T U R E.net to learn why global market leaders like L’Oreal choose Avature to extend the candidate experience. From shoulder taps to first day.
Matt Alder [00:01:23]:
Hi everyone, this is Matt Alder. Welcome to episode 232 of the Recruiting Future podcast. Just over a year ago I interviewed industry consultant Bas Van De Haterdabout his research work exploring emerging assessment technologies. This is a market that’s moving so quickly, I wanted to invite him back a year later to update us on what’s changed. Keep listening for our thoughts on black box algorithms, micro expressions, game based assessments, as well as three case studies of employers achieving considerable success with new assessments technology. Hi Bas, and welcome back to the podcast, Matt.
Bas van de Haterd [00:02:07]:
It’s an absolute pleasure to be back again.
Matt Alder [00:02:10]:
For the very few people out there who may not know who you are, could you just introduce yourself and tell us what you do?
Bas van de Haterd [00:02:16]:
Well, my name is Bas Van De Haterd. I’m a Dutch. I’m a typical slash worker. I do a lot of things. I organize a couple of events in the Netherlands, one international and two in Dutch. I advise companies on how to, for example, assessment tools and I’m a public speaker all over the world. I just got back from Dubai in training the locals on recruiting technology there.
Matt Alder [00:02:45]:
Fantastic. I’m sure the weather difference between Dubai and the Netherlands is probably quite remarkable.
Bas van de Haterd [00:02:53]:
Yeah, it’s 26 degrees C difference, so yeah.
Matt Alder [00:03:00]:
So the last time you were on the show was last September and we were talking about of the work that you are doing around assessments. So assessments are changing. There’s lots of really interesting technology out there and some great case studies and you’ve sort of really made it your mission over the last few years to, you know, get under the skin and find out what’s really happening. So, really interested to get an update on your findings, but. But I think before we do, we probably just need to set the scene again about, you know, why assessment’s changing and why we’re talking about it. So what’s wrong with the way that we currently assess in recruiting or we’ve traditionally assessed in recruiting?
Bas van de Haterd [00:03:42]:
Well, the way we traditionally hire people is we start with a resume which has no predictive value whatsoever, because a resume tells me what you’ve done for who and how long. And the only two things which actually are predictive is how well you did a job and under what circumstances you did well or not well. So we start with deselecting a lot of great talents and selecting people we think have talent. It’s mostly the bias. For some people, if you’ve worked at Royal bank of Scotland, it’s a positive, for others is a negative. The truth is we don’t know how well you did there. So basically, if you’re looking at the way we’re selecting people and we’re only taking the last person usually or the last two or three, to an assessment center, if we do that at all, which means that we are not using the most predictive value early on in the process. And now with modern technology, we have a chance to actually be able to select from the entire pool of candidates the best qualified and then go into the process.
Matt Alder [00:04:50]:
So what should we be measuring in the recruitment process? How do we know if someone’s going to be a good fit for the job and really perform well?
Bas van de Haterd [00:05:01]:
Well, start with exactly what you’ve just said, a good fit for the job. Start with actually defining what are the main characteristics of this job, what are the main items that somebody needs to do, what are the main cognitive traits, psychometric traits, and actually measure that up front and stop by making assumptions that if somebody did a job, that he or she is able to do the job. Because, let’s be honest, we’ve all worked with a lot of people who are completely incompetent, but had been doing the job for them for a decade or over.
Matt Alder [00:05:35]:
Absolutely. So in the sort of process that you’ve been going through in terms of trying to make sense of the evolving world of assessment, because there are obviously, you know, lots of technology providers in this space, lots of, lots of new thinking flying around, lots of sort of potential disruption. What have you been looking at and what, what were your sort of, what were your assumptions going into, going into this process?
Bas van de Haterd [00:06:01]:
Well, my assumptions were that it was eventually quickly a lot better and despite the fact that there’s a lot better than the average resume looking and interview and actually questionnaire based assessment. Because I’m looking at everything except the questionnaire based assessments or I look at those as well but I don’t consider them to be the modern technology, you know, and a questionnaire with a picture which is then called a gamified questionnaire, still not that modern. Most of my assumptions turned out to be correct. Basically it is possible to steer a questionnaire based assessment in a certain way. It’s not possible to make you look perfect, but that’s a different story. And the other thing which is really interesting is a lot of research show that if you’re told in a research no pretend to be the greatest of accountants, people don’t turn out that well as they thought they were. Thing is, do we actually know what makes a great accountant? So my assumptions was also that we actually had an idea of what made somebody good at a job and that one has been blown out of the sky because in most jobs we have no idea, we have no idea about the actual cognitive and mental traits. You need to be good at something.
Matt Alder [00:07:26]:
Absolutely. I mean that’s, that, that, that’s really interesting and I think that that, that speaks to a lot of the issues that companies have with, with recruiting and ultimately with, with performance and retention. So I mean what, you know, what’s sort of changed since we spoke last year? What have you, what have you found out? How’s your thinking changed? Are there now some great case studies around thinking about assessment differently?
Bas van de Haterd [00:07:50]:
Absolutely, absolutely, Matt. Well, the first thing which changed in my perception is I been. I never liked the black box algorithm but I actually thought it might be able to work right now if somebody has a black box algorithm, if a supplier has, I’m running away as hard as I can. One of those companies who does the black box thing is for example, the one that’s most well known and I dare to call them out on it right now because actually the US Senate is calling them out on it right now. It’s called HireVue. And the thing HireVue for example does is they simply, they say listen, give us your best employees. We’ll match all future employees against certain things that the best do as well. But they forget the step in the middle which is alright, so they are doing certain things. We are seeing certain traits or are we actually seeing certain traits? That’s in the higher view because it’s black box. We don’t even know if they put input data, for example, for Micro expression and certain linguistic behavior, they’re not translated back to traits or we don’t know because it’s a black box and we can’t select on the question, is this related to the job or did we have a bias in the first place? And, you know, to give you simple example, I was once in a job interview years and years and years ago, and this woman said to me, oh, it’s so cool. I see on your resume that you do a lot of volunteer work as well, and everybody I’ve interviewed does volunteer work. Yet I didn’t select you people on it. I was like, yeah, you did. But that was unconsciously. Now, there was no reason that altruism would have been a trade for this specific job. It was a marketing manager job. Let’s be honest, you don’t need altruism for that. It was at a staffing firm, and not the most altruistic staffing firm, to be honest. So there was no reason altruism should have been a trade to select on, but there was a personal bias by the HR manager. The thing is that once you start matching all your marketeers with a black box algorithm, the chances are very good that the algorithm will pick up on altruism, which was biased by the hiring manager you had back then or the HR manager in this case. And it’s not a fit for the case, so we need to be able to select those out. That’s the biggest learning I’ve had is the risk of embedding our hiring biases is too big to ever use anything which you can’t. You don’t see the actual traits in the building of the profile.
Matt Alder [00:10:46]:
It’s interesting that you mention micro expressions there because I was actually approached by a journalist on Wired magazine a couple of weeks ago who picked up on the, you know, the current discussion and US Congress thing. Round, round, sort of micro expressions basically wanted me to sort of quote from me saying that it was all. That it was all that they were all bad and they didn’t tell us anything. But I think what you’re saying there is not that. Not that some of the inputs aren’t useful, but actually if you put everything into a black box algorithm where no one actually knows why certain things are being matched to certain other things, that’s. That’s where the danger of bias actually, actually lies.
Bas van de Haterd [00:11:29]:
Absolutely. And now on subject of micro expressions, I do have to admit I’ve tested about four or five of them and the difference between the quality of them is extreme. And I mean, I’ve Tested one which is really, really, really good, which is interestingly the one who says she has the lowest accuracy level and that was by far the best. And I’ve tested some which were just complete and total crap, to be honest. For example, recently there’s been also an academic research, a meta research on all the micro expressions for feelings. So how are you feeling at this moment? Are you nervous? Are you anxious? Those kinds of things. Turned out there was no scientific basis that you could actually measure nervousness from anxiety or lying or stuff like that. Because interestingly enough, we show our state of mind. That was the word I was looking for. And we show our state of mind differently in different circumstances. So if we’re angry, we will have different expressions and micro expressions in a business setting than in a personal setting.
Matt Alder [00:12:51]:
Absolutely. I think, I think it’s interesting that a lot of, a lot of really good science is now coming in to, to challenge some of the assumptions and PR spin that sort of been put out in the past. It’s kind of really interesting that for decades Myers Briggs, Myers Briggs profiles have been held up as a, as a great way of assessing people. And it turns out that effectively they were just made up and there’s no tangible science behind them. But. But yet they still linger on.
Bas van de Haterd [00:13:19]:
Yeah, no, and then the interesting thing, if you talk about Myers Briggs, the scientists have always said it’s a load of bs. Actually recently the Guardian published a letter that Jung wrote before he died saying that his heirs released it, saying that he didn’t know what Myers and Briggs were thinking, but it wasn’t his thinking. Well, Myers Briggs is of course based on Jung, at least that’s what they say. He says it’s not.
Matt Alder [00:13:49]:
No, I mean it’s fascinating stuff. But I suppose before we start geeking out on Jung, Jungian psycho psychoanalysis.
Bas van de Haterd [00:13:59]:
Talk.
Matt Alder [00:14:00]:
Me through what is actually working. What sort of great case studies are there? What sort of. How are people putting these different ways of assessing people into practice and are they getting good results?
Bas van de Haterd [00:14:09]:
Well, the technology I’m currently most looking at, and that’s also because the scientific basis is the best and, and it comes from a different side of science. It comes from the neuroscience, not as much the psychology science. That’s why those two are still very much in debate. But you know, there’s actual neuroscience with MRI scans of your brain and stuff like that as the basis of this is game based assessments. And one of the most amazing case studies I recently came across actually here from the Netherlands is the city of Rotterdam and they’ve decided that they were going to use game based analysis to recruit a recruiter because recruitment wanted to pilot with it. And you need to convince your hiring manager. And if you’re your own hiring manager, that’s much easier. So what they did was they build a profile. What does a good recruiter look like? And all of a sudden you’re having discussions about what kind of trades does he or she need. And to give you an example, one of the traits that they said is very important is technically it’s called attention disconnect. Normal people would call it multitasking. Are you able to quickly switch between tasks? And one of the reasons that’s important for a recruiter at the City of Rotterdam is because at any point you will have between 25 and 30 racks. So you will have a lot of hiring manager complaining at your desk. What are you doing for them and why haven’t you delivered them any candidates? And so you need to really quickly disconnect your attention from one sit to the other and switch really fast between tasks. That was one of the many just, just to make it feasible to give you an example. And what they did was they selected, they gave every applicant that applied to the job the game based assessment. And they had two selection committees, one based on resumes, one based on the tests. So the test guys just said, we want the five best scoring people on the test, you know, and one actually had a perfect score of 100% match with the profile they thought of. The resume guys came up with for the five interviews they were going to do, they had three different candidates. They ended up mixing it up. And the most beautiful part of this case study was they had one perfect resume candidate, perfect experience, all the major brand names, six to eight years experience, what they thought they were looking for, and 100% match on the actual testing, had only six months of experience. So they were like, yeah, this isn’t a junior job. Can we actually, you know, he’s not experienced enough. They did eventually end up interviewing him. And, and as you can imagine from why I’m telling you this case study, he rocked the interview. He was so good. While the guy with all the experience completely flunked. Even the interview committee had also people who really believed in resumes in it. And they hired the guy with not enough experience but perfect test score. He’s been working there for almost a year now and I’ve spoken to many of his colleagues and everybody’s like, yeah, probably the best hire we’ve ever done.
Matt Alder [00:17:37]:
Fantastic stuff. So this really works then this really works.
Bas van de Haterd [00:17:40]:
Another case study I want to share with you and I know you have a lot of listeners in the UK as well. Grant Thornton in the UK actually changed our internship, I think about two years ago. What they said is we’re dropping out. We’re not selecting anymore on your grades, we’re not selecting on schools anymore. Everybody who applies gets to do an aptitude test. And if you’re basically you’re going for an accountant job. So if your accounting skills are good enough, if your math is good enough, it’s good enough. And we don’t care if you’re, you know, from privileged backgrounds and you have a private school education or public school education. We don’t care about your grades because your grades can be influenced by a lot of things which have nothing to do with your actual cognitive abilities. And what happened was that they ended up hiring a lot of interns from neighborhoods that their normal accountants wouldn’t even dare to come. Those people stayed after the internship, did the traineeship, became accountants and actually have now have higher billable hours and have an average churn, which is much lower than the average new hire. So basically they make more money for the company and they’re selecting from a pool of candidates which just isn’t going to be assessed if you look at just the school’s resumes and stuff like that. Can I share a third one? One of them is Dutch air traffic control and that’s the third one where they what do you need to be a really great air traffic controller? Basically you need to really handle stress well. Because let’s be honest, the only moments as an air traffic controller that are critical is when there’s a problem. Turns out that handling stress well is not a prerequisite for any kind of university education. So what they managed to do, they started with there’s no formal education for air traffic controllers, right? I mean the company basically has their own traineeship. They used to start a class of 20 to end up with six because they of course tested, you know, will you be good? Because they started with game based assessments, now they can start with a class of 10 to end up with eight. So the dropout rate and it’s saving millions. But the other beautiful thing about this is they, so they’re using game based assessments to actually measure do you have the traits which we need, you know, also that fast switching but also handle stress well, being able to think, you know, anticipation, think two steps ahead. And what happened was that they ended up dropping every educational demand. They started in the Netherlands. What you need to finish university. Then they said, you need to finish college. And now they’re just like, if you’ve got a high school degree, that’s fine. Because we know, you know, being an air traffic controller is all about learning a book of procedures that’s actually pretty easy to do. And it’s about cognitive traits, which apparently we don’t need to be good at what we call higher education. So what they did was they opened up their entire pool for people who used to, because they didn’t fit in the educational system, weren’t qualified. And now they’re actually taking in everybody who finished high school and say, listen, just do the test and we’ll see if you have the quality to be an air traffic controller.
Matt Alder [00:21:17]:
Absolutely. Absolutely fascinating stuff. And I suppose these case studies kind of lead on to my next question, which was when we were talking about this last September, one of the issues that we kind of identified was the lack of adoption. So lots of interesting technologies out there getting lots of funding and lots of airtime in terms of sort of debate and discussion. But actually the adoption from employers was pretty slow or pretty minimal. How’s that changed in the last 12 months?
Bas van de Haterd [00:21:48]:
It has, it has changed massively, Matt. And to give you an example, one of the Dutch ministries is actually my major client right now. I mean, even what we should be usually perceived as slow governmental organizations are now actually implementing this. And to be honest, I think the adoption rate is even higher. But a lot of them are afraid to talk about it because, for example, Unilever has been using both hirevue as well as biometrics for years and they only get bad press about it. So I think the adoption is actually higher than even. I know. But a couple of major organizations are using this. I know that Arctic Shores has a major client list in the uk. They’re actually looking at the UK military right now. They’ve got some case studies there. The adoption has gone up. What I’ve not yet seen is many excellence case studies where they actually are measuring the quality of hire and things like retention rate, which is interestingly enough, because most companies haven’t defined the quality of hire.
Matt Alder [00:23:10]:
Of course, I think that’s the fundamental problem. As you said earlier, if people don’t actually understand what’s required to do the job and what good looks like that, it’s. It’s very difficult to, to assess anyone for that.
Bas van de Haterd [00:23:22]:
Yeah. But the other part is, Matt, it would be interesting if we actually KN are good at a job. But as soon as I go into an HR department And I ask, you know, do we have any data on who’s actually good at being a recruiter? Most are like, yeah, how should we be measuring good as a recruiter? Or in the case of this ministry, who’s a good policymaker? Let’s start with this debate about what, what defines good policy? I can tell you the next decade, we’re not finished about what is good policy. That’s why we have elections every four years and change policy.
Matt Alder [00:24:00]:
Absolutely. I think, you know, there’s a whole new podcast here about, you know, measuring performance and relating that back to, back to recruiting. Just as a final question for this podcast, what do you think’s next? If we were to sort of chat about this again in 12 months time, what might have changed or what might be on the. On the horizon?
Bas van de Haterd [00:24:22]:
What I hope will be next is that we’ll have much more basic job descriptions already funded in the technology. So we might have a chatbot which actually does linguistic analysis, which is able to select healthcare workers and which is able to select between elderly care worker fit or mental care worker fit, for example. I think people should be working on that. So I assume that if that’s my idea, others will be. And what I think will happen is in a year’s time, there will be a lot more case studies.
Matt Alder [00:25:04]:
Absolutely. Well, let’s hope so. Bas, thank you very much for talking to me.
Bas van de Haterd [00:25:09]:
Thank you for having me. Nat.
Matt Alder [00:25:11]:
My thanks to Bas. You can subscribe to this podcast in Apple Podcasts or via your podcasting app of choice. Please also follow the show on Instagram. You can find it by searching for recruiting future. If you’re a Spotify or Pandora user, you can also listen to the show there. You can find all the past episodes@www.rfpodcast.com. on that site, you can subscribe to the mailing list and find out more about working with me. Thanks very much for listening. I’ll be back next week and I hope you’ll join me.
