If you've applied for a job in the last two years, an AI probably read your resume before a human did. New research suggests it may have judged you on stereotypes it made up on its own.
A paper accepted to ICML by researchers at Princeton and the University of Chicago found that large language models don't just inherit human bias when screening candidates. They generate new stereotypes from scratch, even about demographic groups that don't exist.
The researchers ran an experiment where models made hiring decisions about applicants labeled with fictional groups. No cultural baggage, no training data that could have taught the model who to favor. The models still developed consistent preferences anyway, essentially inventing their own prejudice from thin air.
"LLMs are eager to create generalizations from limited data. That's literally a lot of what they're optimized for," Ryan Liu, a Princeton PhD student and coauthor of the study, told MIT Technology Review.
That eagerness matters because these are the same models sitting between people and jobs. And when they get it wrong, they tend to get it wrong the same way for everyone.
That last part is the finding from a separate Stanford HAI field study tracking 3.4 million real job applicants across 156 employers. Because most companies use the same handful of AI vendors, a candidate rejected by one system tends to get rejected by all of them. About 10% of applicants who submitted four applications were shut out of every job they applied for, a rate higher than random chance would predict.
Stanford's Dan Jurafsky, who worked on the study, said the pattern is arguably worse than the human bias it was supposed to fix. "The AI algorithms we studied were much more likely to act identically, leading a person to be universally rejected, than if the companies were acting independently," he said in an interview about the research.
The scale is what should give any hiring manager pause:
- 90% of US employers use AI screening tools to sort candidates, most relying on the same small group of third-party vendors, according to Stanford HAI.
- 88% of AI vendors cap their own liability, often to monthly subscription fees, while only 17% guarantee regulatory compliance. If something goes wrong, the employer eats the lawsuit.
- 26% of Black applicants and 15% of Asian applicants in Stanford's study applied to positions where the AI actively discriminated against their group.
Regulation exists but barely functions. New York City passed Local Law 144 in 2021, requiring employers to audit hiring AI for bias and notify candidates when it's being used. A state comptroller audit released in December found the law is mostly theatrical. City regulators identified one likely violation across 32 employers reviewed. Independent auditors looking at the same companies found 17. Test calls to NYC's 311 complaint line about AI hiring bias were routed to the wrong agency two-thirds of the time.
Angelina Wang, a Cornell computer scientist who wasn't involved in the ICML study, said the fact that models can invent bias from their own hiring experience is "a really serious implication that they should grapple with." A system that remembers past screenings can start over-indexing on whatever patterns it noticed before, whether or not those patterns actually predict anything about the next candidate.
The pitch for AI hiring was that it would strip the messy human stuff out of the process. Take the gut feeling, the pattern recognition, the lazy shortcuts, and replace them with math. It turns out the math has its own shortcuts, and because everyone bought them from the same vendors, the shortcuts are now industry-wide. The next few years of hiring lawsuits are going to be about who actually owns that mistake, whether it's the employer that deployed the model, the vendor that sold it, or the regulator that never checked. Right now the answer looks a lot like nobody.
