Abstract
Although interaction with mundane automated aids like spelling and grammar checkers is commonplace, it is a surprisingly little studied subject, of which not all characteristics are equally well understood. In a series of experimental studies, we demonstrate how a novel experimental paradigm based on Signal Detection Theory can be used to study a cognitive task, in which different aspects of performance, trust, and confidence of users interacting with an imperfect automated writing aid are tested. Five closely related experiments are reported, in which participants make a series of judgments of which of a pair of similar sentences is better, or whether a single sentence is correct or not. Our experimental hypotheses all derive from the overarching hypothesis that participants will be able to interpret and make use of an automated aid's suggestions and the aid's own estimation of the likelihood of its suggestions being correct.The first, and overriding contribution of this thesis is to begin an experimental exploration of personal beliefs in relation to performance under uncertainty, and with support from an imperfect automated aid in the domain of text writing and editing, and in particular spelling and grammar checking. Especially the measure of bias, as the propensity to accept automated advice, is an essential measure for our studies, and arguably a major novel contribution of the thesis.
The experiments show that trust in similar systems has less of an effect on participants' performance than we anticipated on basis of the literature. This is also true of perceived self-efficacy, although our findings suggest it may play a more important role if the advice from the system is weak and users must be more reliant on their own skills.
We demonstrate that improving the reliability of the aid's advice positively affects users' performance, trust in the aid, and confidence in their own responses, but also that a highly reliable automated aid still gets underused. Throughout the five experiments, we confirmed the above average effect, people's assumption that their own ability is on average higher than that of others, as well as the overconfidence effect, an overestimation of performance if measured as probabilities of success during a task, but less so if measured as an estimate of success-frequency post-task.
Another novel contribution of this series of experiments is the finding that users can recognise how well a system is doing, even if they do not receive any feedback on the system's performance. Users of a more reliable system proved to be more willing to accept the aid's advice, which suggests effects of the reliability and strength of the advice, the latter of which is represented by the system's communicated likelihood estimation. Without receiving feedback about their own performance, users also show they have an awareness of their own performance, which is demonstrated by a higher level of self-reported confidence in correct responses than in incorrect ones.
We believe our research successfully demonstrates opportunities and limitations of using an experimental paradigm based on Signal Detection Theory to explore various aspects of performance, trust, and confidence this domain. We think that our findings will be valuable for future research as well as for the design of automated aids, and that the methods and analyses developed could usefully be transferred to assisted cognitive tasks in other domains.
| Date of Award | 2022 |
|---|---|
| Original language | English |
| Awarding Institution |
|
| Supervisor | James Laird (Supervisor), Stephen Payne (Supervisor), Linda Newnes (Supervisor) & Simon Jones (Supervisor) |