Abstract
The growth in applications of artificial intelligence (AI) has led to an increase in individuals engaging in AI-supported decision-making - or decision making made with support from algorithms. The combination of human and algorithmic advice has strong potential to improve performance in many tasks. However, human decision-makers often show poor use of advice, showing consistent and strong biases in using algorithms. They may either use advice too little, when the algorithms’ performance is strong (algorithm aversion) or become complacent and use the algorithm too much, even when it makes errors (algorithm over-reliance). In this thesis, I aimed to understand how to improve AI supported decision-making by understanding how to debias this use of advice using psychological interventions. I focus particularly on calibration - or a decision-maker’s ability to use algorithmic advice appropriately, using the advice more when it outperforms the decision-maker, but less when the decision-maker is superior. Calibration can be measured on both the macro-level (across all decisions) and the micro-level (from case to case). There were three key aims of this work; to formulate and validate the framework of calibration; to find effective methods to improve macro-level calibration; and to find effective methods to improve micro-level calibration.
Chapter 1 introduces and reviews previous work on advice taking from algorithms and introduces calibration. Across two experiments in Chapter 2, we measured and assessed calibration, finding evidence that we can reliably separate macro- and micro-level calibration, and that participants have the ability to calibrate on the micro-level. However, three interventions using different types of feedback did not improve calibration or performance. Chapter 3 simulates the effectiveness of a number of different advice-taking strategies, or ’rules’ for taking advice. We find that most strategies would improve the use of algorithmic advice, but that strategies that approximate micro-level calibration perform best. In Chapter 4, we use a selection of these advice-taking strategies as ’meta-advice’, providing users with different guidance on how to optimally weight the algorithmic advice. We find that two strategies - a micro-level strategy, and a meso-level strategy - are able to improve micro-level calibration, but did not result in improved performance. Chapter 5 discusses the key findings of this work, focusing especially on the impacts of the calibration framework on advice-taking research and its implications for the design of decision support systems used in AI-supported decision-making.
| Date of Award | 22 Jul 2026 |
|---|---|
| Original language | English |
| Awarding Institution |
|
| Sponsors | Civica |
| Supervisor | Janina Hoffmann (Supervisor), Tom Fincham Haines (Supervisor) & Thomas Schultze-Gerlach (Supervisor) |
Keywords
- alternative format
- advice-taking
- calibration
- algorithm aversion
- human-ai interaction
Cite this
- Standard