Toto-Rex's story
Toto-Rex gets a temporary job in HR. The new hiring tool ranks 200 applicants in seconds.
Toto-Rex's task: check the top ten, then click 'Approve'. All ten went to the same three universities.
Toto-Rex's manager says: 'Relax. A human checks everything. That's you.'
Toto-Rex's story is made up. The studies and cases in this video are real.
The short answer
Not necessarily. But overruling a tool is not automatically better either.
The details
A University of Washington-led team ran a CV-screening experiment with 528 participants.
Participants chose candidates with and without simulated AI recommendations.
Without AI, they showed no statistically significant racial preference in this experiment.
In some conditions, the AI strongly favoured one racial group.
Participants picked three of five candidates each time.
In the strongest-bias conditions for high-status jobs, the estimated chance of choosing a majority of candidates from the AI-favoured racial group reached about 90%.
People followed the AI whichever group it favoured.
The AI still swayed some people who rated its recommendations as poor quality.
The study also tested an unconscious-bias test.
People who took it before screening, rather than after, showed a trend towards less stereotyped choices.
That difference was not statistically significant in the detailed comparisons.
A different question: is overruling a tool always better?
A different study examined hiring tests at 15 firms employing low-skilled service workers.
Managers who more often went against the test's recommendations tended to hire workers who left sooner.
This held even after accounting for differences in applicant pools.
The study compared managers' choices. It was not a randomised experiment.
This finding concerns how long hires stayed. It does not show whether the test was fair across demographic groups.
Separately, some laws add their own checks.
EU: the AI Act generally treats AI designed to recruit or select candidates as high-risk.
Its main rules start on 2 December 2027, with transitional rules for existing systems.
New York City: employers and employment agencies using covered hiring or promotion tools must make sure the tool had an independent bias audit within the past year.
They must also publish a summary of the results and give required notices.
The law requires the audit, not any particular fix. Other anti-discrimination laws still apply.
Useful to know: neither people nor tools are bias-free by default. How they are checked, and how they work together, matters.
Real-life examples
Real life: how the University of Washington-led experiment worked.
In the AI conditions, each task showed a job description, five CVs, and the AI's recommendations.
Participants had four minutes for each task, to reflect the time pressure of real CV screening.
Names and membership of affinity groups on the CVs signalled each candidate's race.
The jobs ranged from higher-paid roles, such as nurse practitioner, to lower-paid roles, such as housekeeper.
Real life: Amazon's experimental tool.
In 2018, Reuters reported that Amazon had scrapped an experimental AI tool that rated job candidates from one to five stars.
Reuters reported that the team had been building programs to review job applicants' CVs since 2014.
The models were taught to recognise some 50,000 terms from past candidates' CVs.
According to people familiar with the project, it penalised CVs that included the word 'women's' and downgraded graduates of two all- women's colleges.
In effect, Reuters reported, the system taught itself that male candidates were preferable.
Those people said recruiters looked at its recommendations but never relied on them alone.
Amazon said the tool was never used by its recruiters to evaluate candidates.
Real life: New York City's audit law in practice.
In a 2024 study, 155 student investigators checked 391 employers for the audit reports and notices the city's law requires.
They found 18 audit reports and 13 transparency notices.
The researchers said this does not prove employers broke the law, because employers largely decide for themselves whether their tool is covered.
In December 2025, New York State's Comptroller found the city's enforcement system ineffective.
The city agency had received only two complaints about these tools between July 2023 and June 2025.
Back to Toto-Rex
In the experiment, people often followed biased recommendations. So Toto-Rex's click is not a safeguard on its own.
But overruling the tool every time is not automatically better either.
Toto-Rex can ask three questions: who does the tool favour, has it been tested for bias, and who can change its output?
Summary
In short, people often followed a biased AI.
And overruling a tool is not automatically better either.
Neither people nor tools are bias-free by default.
What matters is how they are checked and how they work.
Three key terms
Automation bias
The tendency to rely too heavily on automated output.
People may follow it even when other information says otherwise.
For example: picking the AI's favourite candidate without checking.
Human in the loop
A person takes part in a decision that involves AI. Taking part does not guarantee a fair result.
Bias audit
Under New York City's law, an independent auditor evaluates a covered tool.
The auditor compares outcomes across sex, race or ethnicity, and combined groups.
New York City requires one within the year before a covered hiring or promotion tool is used.
Further reading
Start with the University of Washington-led paper. It shows which differences were statistically significant.
Then 'Discretion in Hiring', the study of job tests at 15 firms.
For the New York City rules, see the regulator's FAQ on automated employment decision tools.
For automation bias in general, see the CSET report. For the Amazon case, see Reuters' report of October 2018.
Sources
Wilson, Sim, Gueorguieva, and Caliskan, 'No Thoughts Just AI: Biased LLM Hiring Recommendations Alter Human Decision Making and Limit Human Autonomy', AIES, 2025.
Hoffman, Kahn, and Li, 'Discretion in Hiring', Quarterly Journal of Economics, 2018. Kahn, Probasco, and Kinoshita, 'AI Safety and Automation Bias: The Downside of Human-in-the- Loop', Center for Security and Emerging Technology, November 2024.
EU AI Act (Regulation (EU) 2024/1689), Articles 6(2) and (3), 111(2), and 113, and Annex III, point 4(a), as amended by Regulation (EU) 2026/1744. New York City Administrative Code, sections 20-870 and 20-871 (Local Law 144 of 2021). 6 RCNY sections 5-300 to 5-304.
DCWP, Automated Employment Decision Tools FAQ.
Reuters, 'Amazon scraps secret AI recruiting tool that showed bias against women', October 2018.
Wright and others, 'Null Compliance: NYC Local Law 144 and the Challenges of Algorithm Accountability', FAccT 2024. Office of the New York State Comptroller, 'Enforcement of Local Law 144', 2 December 2025.