Anyone familiar with HR practices probably knows of the decades of studies showing that résumé with Black- and/or female-presenting names at the top get fewer callbacks and interviews than those with white- and/or male-presenting names—even if the rest of the résumé is identical. A new study shows those same kinds of biases also show up when large language models are used to evaluate résumés instead of humans.
In a new paper published during last month's AAAI/ACM Conference on AI, Ethics and Society, two University of Washington researchers ran hundreds of publicly available résumés and job descriptions through three different Massive Text Embedding (MTE) models. These models—based on the Mistal-7B LLM—had each been fine-tuned with slightly different sets of data to improve on the base LLM's abilities in "representational tasks including document retrieval, classification, and clustering," according to the researchers, and had achieved "state-of-the-art performance" in the MTEB benchmark.
Rather than asking for precise term matches from the job description or evaluating via a prompt (e.g., "does this résumé fit the job description?"), the researchers used the MTEs to generate embedded relevance scores for each résumé and job description pairing. To measure potential bias, the résuméwere first run through the MTEs without any names (to check for reliability) and were then run again with various names that achieved high racial and gender "distinctiveness scores" based on their actual use across groups in the general population. The top 10 percent of résumés that the MTEs judged as most similar for each job description were then analyzed to see if the names for any race or gender groups were chosen at higher or lower rates than expected.
A consistent pattern
Across more than three million résumé and job description comparisons, some pretty clear biases appeared. In all three MTE models, white names were preferred in a full 85.1 percent of the conducted tests, compared to Black names being preferred in just 8.6 percent (the remainder showed score differences close enough to zero to be judged insignificant). When it came to gendered names, the male name was preferred in 51.9 percent of tests, compared to 11.1 percent where the female name was preferred. The results could be even clearer in "intersectional" comparisons involving both race and gender; Black male names were preferred to white male names in "0% of bias tests," the researchers wrote.
These trends were consistent across job descriptions, regardless of any societal patterns for the gender and/or racial split of that job in the real world. That suggests to the researchers that this kind of bias is "a consequence of default model preferences rather than occupational patterns learned during training." The models seem to treat "masculine and White concepts... as the 'default' value... with other identities diverging from this rather than a set of equally distinct alternatives," according to the researchers.
The preference shown by these models toward or against any one group in each test was often quite small. The measured "percentage difference in screening advantage" was around 5 percent or lower in the vast majority of comparisons, which is smaller than the differential preference rates shown by many human recruiters in other studies. Still, the overwhelming consistency of the MTEs' preference toward white and/or male names across the tests adds up across many different job descriptions and roles.
The results in this controlled study might also not match how recruiters use AI tools in the real world. A Salesforce spokesperson told Geekwire that "any models offered for production use go through rigorous testing for toxicity and bias before they’re released, and our AI offerings include guardrails and controls to protect customer data and prevent harmful outputs."
That kind of protection can be important, as Amazon learned in 2018 when it was forced to scrap an internal AI recruiting tool that showed a bias against women. And research like this can be important to counteract the sci-fi-aided impression among some users that AIs are incorruptible machines that can't be swayed by any undue influence. On the contrary, in the real world, AI bots quite often reflect the negative biases that can be inherent in training data.
