Human-in-the-Loop Oversight of AI is Compromised by Political Preferences

Humans are legally responsible for monitoring sensitive Artificial Intelligence (AI) systems, requiring a human-in-the-loop (HITL) to have the final say in approving or rejecting AI decisions. However, behavioral science shows humans are not error and bias free. We report the results of three experiments (n = 5798, 115960 decisions) modeled on real-world welfare allocation scenarios in which locals and immigrants are considered for financial support. Results revealed that HITL oversight does not reliably correct algorithmic errors, and may even exacerbate errors. Moreover, political preferences of the HITL shape the type of errors they make. Interventions incentivizing accuracy or clarifying instructions reduced HITL error rates. Nevertheless, HITL error rates were higher than algorithmic error rates. These results point out that HITL is not a silver-bullet solution across settings. Our work calls for more systematic research into when, and how, oversight of AI can effectively reduce error and improve decision making quality.