Help Wanted! The Truth About Talent Assessment & Hiring Tech
Psych Tech @ Work
LLMs, Talent Assessments, & Hiring- Research Meets Practice
0:00
-1:02:17

LLMs, Talent Assessments, & Hiring- Research Meets Practice

"We’re generating assessments faster than ever, but our real test is ensuring that these tools are fair and reliable across diverse candidate groups."

–Louis Hickman

In this episode I welcome my friend, super dad, and ex- professional wrestler Louis Hickman for a killer conversation about the ins and outs of using LLMs to create and score assessments.

Louis is a professor at Virginia Tech specializing in research on AI and large language models in assessment and hiring processes.  He knows a thing or two about this stuff and we waste no time tackling some really great topics centering around the cutting edge of research and practice on the subject of LLMs and assessments.

This is a must listen episode for anyone developing, or considering developing, LLM based assessments. Or anyone who wants to educate themselves about how LLMs behave when asked to be I/O psychologists.

Topics Covered:

  • LLMs in Assessment Center Role-Plays:

    • Using LLMs to simulate realistic role-play scenarios for assessments, with the challenge of ensuring consistent, replicable candidate experiences.

  • Evaluating Open-Ended Text with LLMs:

    • How LLMs score open-ended responses and the observed biases, especially when diversity prompts only partially reduce disparities.

  • Consistency in AI Scoring:

    • Ensuring LLMs apply scoring criteria consistently across diverse candidates and settings.

  • Applicant Reactions to AI Interviews:

    • How candidates perceive AI-driven interviews, with many expressing discomfort due to the perceived inability to influence AI decisions compared to human interactions.

  • Predicting Responses to Assessment Items:

    • The potential for LLMs to predict candidate responses without actual data, though accuracy remains limited by model training and inherent biases.

  • Impact on Academic Research:

    • LLMs' influence on research publications, with concerns over AI tools favoring self-generated content and potentially amplifying biases in academic discourse.

Listen to the episode to hear the skinny on these topics and more!

And of course we have fun with this episode’s “Take it or Leave it” articles.

Article 1 

The Impact of Generative AI on Labor Market Matching.” An MIT Exploration of Generative AI”, 

explores the use of LLMs on matching job seekers and employers.

Article 2

Four Singularities for Research: The Rise of AI is Creating Both Crisis and Opportunity

In this article from Ethan Mollick’s Substack blog One Useful Thing discusses the positive and negative impact of LLMs on academic research.

Discussion about this podcast

Help Wanted! The Truth About Talent Assessment & Hiring Tech
Psych Tech @ Work
Science 4-Hire is now Psych Tech @ Work! - a podcast about safe innovation at the intersection of psychological science, technology, and the future of work.
Psych Tech @ Work promotes safe technological innovation and human/machine partnerships as an essential force in creating equilibrium and between psychology and commerce.  Maintaining this balance in a time of unprecedented change is essential for ensuring that the future of work is ethical, positive, and prosperous.  
Creating such a future requires an unprecedented level of interdisciplinary collaboration.  With the goal of educating, engaging, and inspiring others through thoughtful and practical discussions with guests from a wide variety of backgrounds and specialties, Psych Tech @ Work provides a smorgasbord of food for thought and practical takeaways about the issues that will make or break the future of work!