The explosion of AI-generated applications isn’t just breaking traditional recruiting – it’s creating an unprecedented opportunity by making sophisticated assessment tools accessible for early-stage screening. There is now the opportunity to filter thousands of applicants based on actual predictive data. However, the vendor landscape here can be confusing, and some offerings lack the transparency that employers need.
So how can organizations identify tools that leverage AI’s efficiency while respecting established peer-reviewed assessment science?
My guest this week is Djurre Holtrop, Assistant Professor at Tilburg University. In our conversation, Jura reveals how AI could democratize evidence-based assessment for organizations of all sizes and offers advice on best practices and the future assessment landscape.
In the interview, we discuss:
• The science-practice gap in assessment
• The tension arising from the rapid development of AI and the need to evaluate work performance results over time.
• AI isn’t making the process worse because CVs and cover letters have no predictive validity.
• How is AI improving the science?
• Transforming recruiting with early-stage screening
• How bias carries through the recruiting process
• Resetting our mental models
• What does the future look like?
Recruiting Future helps Talent Acquisition teams drive measurable impact by developing strategic capability in Foresight, Influence, Talent, and Technology.
If you’re interested in finding out how your TA function measures up in these four critical areas, I’ve created the free FITT for the Future Assessment. It’ll give you personalised insights to help you build strategic clarity and drive greater impact immediately.
Just head to mattalder.me/podcast to complete the assessment—it only takes a few minutes.
Follow this podcast on Apple Podcasts.
Follow this podcast on Spotify.
00:00
Matt Alder
Jobseekers Use of AI is creating a recruiting crisis with rapidly increasing volumes of applications, all of whom appear to have the perfect resume for the job. But could AI actually be the solution to its own problem by democratizing access to proven science that can predict future performance and moving it to the top of the recruiting funnel? Keep listening to find out.
00:29
Matt Alder
Support for this podcast comes from smart recruiters. Are you looking to supercharge your hiring? Meet Winston Smart Recruiters AI Powered Companion. I’ve had a demo of Winston. The capabilities are extremely powerful and it’s been crafted to elevate hiring to a whole new level. This AI sidekick goes beyond the usual assistant handling all the time consuming admin work so you can focus on connecting with top talent and making better hiring decisions. From screening candidates to scheduling interviews, Winston manages it all with AI precision, keeping the hiring process fast, smart and effective. Head over to smartrecruiters.com and see how Winston can deliver superhuman results.
01:34
Matt Alder
Hi there. Welcome to episode 726 of Recruiting Future with me, Matt Alder. Recruiting Future helps talent acquisition teams drive measurable impact by developing their strategic capability in foresight, influence talent and technology. If you’re interested in finding out how your TA function measures up in these four critical areas, I’ve created the free Fit for the assessment. It’ll give you personalized insights to help you build strategic clarity and drive greater impact immediately. Just head over to Mattalder.me/podcast to complete the assessment. It only takes a few minutes. This episode is about talent and technology. The explosion of AI generated applications isn’t just breaking traditional recruiting, it’s creating an unprecedented opportunity by making sophisticated assessment tools accessible for early stage screening. There is now the opportunity to filter thousands of applications based on actual predictive data.
02:44
Matt Alder
However, the vendor landscape here can be confusing and some offerings lack the transparency that employers need. So how can organizations identify tools that leverage AI efficiently while respecting established peer reviewed assessment science? My guest this week is Djurre Holtrop, assistant professor at Tilburg University. In our conversation, Djurre reveals how AI can democratize evidence based assessement for organizations of all sizes, and offers advice on best practises and the future assessement landscape.
03:24
Djurre Holtrop
Thank you, hi. Really good to be here.
03:27
Matt Alder
Pleasure to have you on the show. Please could you introduce yourself and tell everyone what you do?
03:32
Djurre Holtrop
I’m Djurre Holtrop. I work at Tilburg University as an Assistant professor, meaning That I do teaching and research. I specialize in personnel selection recruitment and I think that’s what we’re going to talk about today. I have a background in assessments, so I started out in practice with a consultancy, a test developer and then sort of, yeah, got into the discipline and learned a lot and got really passionate for personnel selection over time. And yeah, then I started to do research.
04:05
Matt Alder
Fantastic. And kind of really interested to sort of talk about your research and all that kind of stuff as we sort of move through.
04:11
Matt Alder
To start with though, I’ve got a.
04:12
Matt Alder
Question because the more people that I’ve talked to about assessment, the more that I’ve looked into it and all of this kind of stuff, there is this kind of real gap between the science, the peer reviewed science of what works and actually what many employers actually do in practice when it comes to sort of assessing people. What do you think causes this gap?
04:33
Djurre Holtrop
That’s an excellent question to start with. You know, if I had the answer, I think that would be amazing. But many people have discussed this. Many academics have been frustrated with that practice is not picking up on what they preach are the best solutions. And they have attempted to convince organizations to just work in the evidence based ways that have been discovered, researched. So I’m not going to say that I have the answer for what is causing this gap, but I do see that there is this gap between science and practice. And I think also that the answer is not simple. There are many layers to this issue and organizations are complex with lots of people that you need to convince when you’re implementing evidence based practices. That’s one side of the coin.
05:30
Djurre Holtrop
I think the other side of the coin is that to an extent science might be oversimplifying the problems that exist in practice. And that might also lead to sort of like this, the fact that knowledge is then harder to translate to practice. And what do you pick up from this? So, for example, scientists use huge studies called meta analyses where we summarize hundreds of studies in one big study and then we conclude that certain personality factors such as conscientiousness, are predictive of job performance. Therefore, we say, all right, conscientiousness is useful for selection purposes, but we also know that every job is different, every team is different, every organization is different, and that you should really do a job analysis to determine which characteristics are important for a specific job.
06:27
Djurre Holtrop
So the meta analyses that academics hold incredibly high regard are also at the same time a little bit uninformative if you’re in practice, because they don’t necessarily apply to the job that you’re recruiting for at the moment.
06:43
Matt Alder
Yeah, I think that makes sense. And I think also there’s a lot of stuff about how assessment is kind of marketed and bought and sold and all those kind of things as well, I think. So we’re at a really interesting time at the moment because AI really does have the potential to transform how we look for talent, how we assess talent, or certainly allow us to do things at scale that we’ve never been able to do before. How is that coming out in assessment at the moment? I mean, what does the landscape for AI and assessment look like? What’s changing? What’s surprising people? What’s hype, and what’s just wrong?
07:19
Djurre Holtrop
Yeah. Again, this is one of those questions where my answer is probably going to be outdated in a week from now. That’s fantastic. Right. But I think that’s the number one thing that I’m taking away is it’s going at such a breakneck speed that it’s hard to keep track of. Solutions are popping up left, right, and center. And one of the things that I’m seeing is that while we’re moving at such a high speed, at the same time, we’re sometimes forgetting all these best practices and these things that we’ve learned over the past century of trying to understand recruitment and selection. Right. So sometimes I’m seeing solutions that are almost resembling phrenology, where you like, we’re looking at how people look in certain ways and then assessing them based on that.
08:16
Djurre Holtrop
I’m not for those types of solutions, but I also know that we’ve been there already before the technology was allowing us to do this and also concluded that it doesn’t add much, basically adds bias. So sometimes I’m worried that having technology is just almost an excuse to make the same mistake again. And that’s something to be wary of. But at the same time, vendors are really competing with each other and trying to carve out their space in the market, so they need to move fast. And that is really at odds with proving that your tool is able to select the best employees. Because we all know that it takes time to evaluate the effectiveness of a tool. I find that this creates this tension between the speed the technology is now moving at and at same time, evaluating the effectiveness of our tools.
09:11
Djurre Holtrop
Usually you want people to perform for a year or so, and then you can say, all right, were actually accurate about saying that this was going to be a good employee. So how do you put those two together? That’s something that I Think we need to start to resolve.
09:27
Matt Alder
Yeah, 100%. And I think that, you know, with this sort of gap that were talking about a second ago, there’s also a danger that vendors who come onto the market trying to reinvent the wheel, they’re trying to sort of invent things that already exist or they’re not properly paying attention to existing science and using pseudoscience and other crazy things. And it’s. Yeah, there’s a. There’s a lot of potential risk out there, isn’t there?
09:49
Djurre Holtrop
Yeah, yeah, absolutely. As an organization, this is asking a lot from you too, right? So you need to understand technology, you need to understand probably psychometrics, too, to be able to evaluate if a tool is working and can deliver and to distinguish pseudoscience from real science. And I find that really difficult personally. It’s my profession, it’s my specialism to figure these things out. And I sometimes struggle. So I can only imagine that people who don’t have the background and the training that I do, how much they would struggle to even understand the information that is out there and evaluate a tool.
10:28
Matt Alder
Yeah, I mean, have you got any advice to people on that? Are there any shortcuts, any specific things that they should be looking for or questions that they should be asking?
10:36
Djurre Holtrop
One thing is that there are a few basics. Always check if the tool measures the competencies that you’re interested in. Does it measure them reliably and are the scores predictive of future performance? Those are probably the big three, but these also apply to traditional assessments. What I’m saying is don’t deviate too much from how you would look at a traditional assessment, and don’t be blinded by all the glamorous elements of the novel tools. If you feel like you cannot do this, just consult an independent expert. I think that would be perfectly fine. In your purchasing. You can consult experts. There are plenty of experts out there who will be able to advise you on if at all is. Yeah. Of high quality or not.
11:32
Matt Alder
Yeah, no, 100%. I think it’s. It’s an area where people should really lean on some of the great expertise that is out there. Just in terms of, I suppose, the work that you do. What impact could AI have on the actual science? Is it. Are there things that it can do quicker? Is it making you rethink how everything works? What’s the sort of the. The impact of the new technology on, you know, on the thinking behind all of this?
11:57
Djurre Holtrop
I think in two ways, AI can really contribute. One is efficiency. So we can do the same things faster and sometimes with a little bit less human error, but mostly faster. And that’s great for science, that’s also great for the high pressure industry that we’re in. But on the other hand, I think it also opens up new possibilities in terms of the types of data that we can access and work with. But that’s the part where we need to be careful. Right. So the data that AI can unlock for us is that interesting data. So don’t just use data for the fact that it’s there and that you can unlock it, but think about the relevance of the data. And I think that is the part where we really need to pay attention. Do we unlock relevant data with the tools?
12:51
Matt Alder
Is your TA function fit for the future? Over the last 10 years I’ve done podcast interviews with hundreds of the world’s most successful TA leaders and discovered exactly what they all have in common. Four critical capabilities that drive their strategic foresight, influence, talent and technology. I’ve created a free assessment specifically designed for busy TA leaders like you to benchmark your capabilities against these four essential areas. You’ll receive clear insights into your current strengths and opportunities with practical guidance on enhancing your strategic impact. The assessment is completely free, takes 5 minutes to complete and you get your results instantly. Benchmark your talent acquisition capability right now and start proactively shaping your future. Visit Mataulder Me Podcast. That’s matalder Me Podcast.
13:59
Matt Alder
Technology moves at a million miles an hour, but you know, this is very human based and humans don’t move that quickly. So again, presumably it kind of takes time to kind of evaluate this stuff.
14:11
Djurre Holtrop
Yeah. Yes, you do see that everybody is trying to jump on the bandwagon applicants too. Right. So they are also jumping on the bandwagon and using AI now with the introduction of large language models. In a way it’s a fantastic. It’s become a fantastic arms race that I’m looking at with fascination, to be honest. Where applicants are now and people organizations are also developing tools for applicants. Right. So first it was just LLMs and now people are building things on top of those LLMs that applicants can use. Yeah, that makes this a whole new era of recruitment in my mind.
14:53
Matt Alder
Yeah, I agree with you 100%. I think it’s not quite evenly spread at the moment, but I know many organizations are kind of getting overrun with applications particularly. It’s also the contributing factor of the economy in many countries. But the AI driven applications is such a big thing. I mean from a personal perspective, I’m kind of with you on this, I think it has the potential to break the recruiting process and leave us very much in need of a new one. Where do you think assessment kind of fits in terms of being able to sort of fix this problem where you’ve just got, you know, loads of perfect applications coming in using, you know, CVs and the older, the old school way of doing things and it just becomes very difficult for employees to make judgments and move forward.
15:40
Djurre Holtrop
So the easy targets for AI applicants using AI were CVS and cover letters. Right. And I think you would be mad not to use an AI at the moment when you’re writing your cover letter, in my mind.
15:57
Matt Alder
But also I think this is the other thing. It’s like most of the writing tools that people use have AI built into them. So whether it’s, you know, so it’s.
16:05
Matt Alder
Kind of, you, most can’t, you almost.
16:07
Matt Alder
Can’T avoid it when you’re writing that kind of thing.
16:09
Djurre Holtrop
Agreed. And I think that’s not really a problem one, because in my mind, CVS and cover letters have no proven validity to them. So this is not worsening the process, it’s worsening the volume for sure. But the decisions that be made based on cover letters and CVs par for looking then for essential credentials that you need to have to be able to do the job, like you need a driver’s license if you want to drive a car. Except for that people were reading much too much into these sources of information. I hope personally that we’ll move past them now finally, after there’s been a century of no proof that this works, and now start to look for different assessments that we can use for early stage screening. Right.
17:07
Djurre Holtrop
Because that’s what we’re talking about, early stage screening in an effective way where we can distinguish skills that people have and fit people with the job.
17:16
Matt Alder
And do you see any evidence of that already happening or is it going to just take a while for people to sort of realize that might be the solution?
17:25
Djurre Holtrop
You know, those tools have always been out there, even if they were like the traditional questionnaires could have replaced a CV and a cover letter because how long did it take you to write a cover letter back in the day? Probably half an hour or so, maybe a bit longer for some people completing such a questionnaire about the same length. So we could have done that already. I’m just hoping that this will push organizations to really start innovating and moving in different directions that are more valid. Also, what I’m seeing, like I’m Getting signals of organizations saying, all right, so now cover letters and CVEs are all becoming very samey homogeneous. So what do we do now? Can we talk now about early stage assessments and questionnaires or maybe those types of video interviews to just make it a bit harder to use AI?
18:26
Djurre Holtrop
And I know that there is AI tooling also available for applicants that helps them to complete these types of assessments, but it’s less accessible and fewer people are using that now with the applicants.
18:41
Matt Alder
Yeah, I mean, that makes sense. And I suppose also, you know, what are the implications for the rest of the recruitment process? So if we’re moving to a model where we’re getting much better data right at the start of the process, how does that kind of affect the way kind of recruiting moves, the process moves forward from there?
19:03
Djurre Holtrop
This is probably good news for fairness in hiring decisions. What we’ve seen is that cover letters and CVs tend to be decisions based on those tend to be biased. And we also know that early stage bias even masks bias later in the process. So what I mean is if you have a lot of bias in the early stages of your selection, then the later stages of your selection will always seem fair because you’ve sort of let the same people through anyway and discrimination has already happened. So what I’m hoping is that this will make sure like this creates a possibility for early stage screening to be fairer and then hopefully that will allow later stage screening to also be evaluated in that manner.
19:51
Matt Alder
Yeah, that makes sense actually. And I think that there must be so many organizations who do that, who think they have a very unbiased process, but actually the bias is sitting there right at the, very, right at that very first stage that they’re not thinking about. So how do you think this kind of pans out? Just, you know, I know that you’re obviously an evidence based scientist, but just for now projecting to the future, where do you think this might take us in terms of how recruiting works as things develop? If were sort of looking at this, in three or four years time.
20:23
Djurre Holtrop
The differences between organizations are going to be magnified. So whereas small medium businesses, they won’t move as fast because tacking on to those technological innovations requires a bit of resources. Right. And it requires changing practices that you’ve, you already, that have become traditions. So it’s very habitual recruitment in my mind and those habits have been firmly established. But larger organizations are looking to innovate and they also have. The problems with the volume of recruitment are more pronounced there for the larger organizations. So I feel as if this will create a bigger divide between larger and smaller organizations in terms of how they deal with recruitment.
21:11
Matt Alder
I think that’s very true because it’s always. I mean, there’s two things there that kind of really strike me. I think that whole idea of tradition is just so strong because as you say, in some ways things haven’t changed for 100 years. Also, everyone who has a job has been for a recruitment process. Some very sort of set views about what it should. What it should be or what it should look like. And that’s proven to be very sort of difficult to kind of break in the past. But. But also I think that point there about the larger organizations with their. Their bigger resources, but also their kind of bigger throughput of recruiting, you know, compared to the smaller organizations who don’t have the technology and are kind of likely to be stuck in that kind of traditional way of doing things.
21:53
Matt Alder
So that’s quite a. Quite a significant issue there, isn’t it?
21:57
Djurre Holtrop
Yeah, I think so, too. Large organizations will have the specialists to innovate first. Right. And then I suppose small medium enterprises will start looking at what some large organizations are doing and maybe imitate that innovation to an extent. We’ll see. What I also think then, in terms of another trend is that not only in recruitment, but also at work, AI is playing a bigger and bigger role. Right. So we need to also start considering how much do we want applicants to use AI in the application process? Because using AI may be an indication of how. In the application process, maybe an indication of how well they can use AI at work.
22:46
Djurre Holtrop
So as the roles, the positions that we have change and the demands that we have for our workers, then we should also look at the recruitment process and make sure that these are synergetic so that the recruitment process mirrors what we do on the workflow. And thinking about how we ask applicants to use AI, I think that is also a big question. So at the moment we should. So we can also decide to embrace the fact that applicants are using AI, but instead of saying, all right, funnel those efforts a little bit so that we can take information away from it to make sure that we understand what type of worker they will be. And I think that is also something that we still need to.
23:32
Matt Alder
I think it’s definitely going to be an interesting few years. Djurre, thank you very much for talking to me.
23:39
Djurre Holtrop
Yeah, no worries. That was a really lovely conversation.
23:43
Matt Alder
My thanks to Djurre. Don’t forget, if you haven’t already. You can benchmark your talent acquisition capability quickly and easily by completing the free Fit for the Future assessment. Just head over to Mattalder.me/podcast. It only takes a few minutes and you’ll receive valuable insights right away. You can follow this podcast on Apple Podcasts, on Spotify, or wherever you listen to your podcasts. You can also search through all the past episodes at recruitingfuture.com on that site. You can also subscribe to our weekly newsletter, Recruiting Future Feast, and get the inside track on everything that’s coming up on the show. Thanks very much for listening. I’ll be back next time and I hope you’ll join me.
