One Is All, All Is One:

: Cross-Modal Displays for Inclusive Design and Technology

Abstract

Sensory substitution phenomena transform the representation of one sensory form into an equivalent from a different sensory origin. For example, a visual feed from a camera can be turned into something that can be touched or sounds that can be heard. The immediate applications of this can be seen in developing assistive technologies that aid vestibular problems, and visual and hearing impairments. This raises the question of whether perception with sensory substitution is processed like an image, or like a surface, or a sound. Sensory substitution techniques offer a great opportunity to dissociate the stimulus, the task and sensory modality, and thus provide a novel way to explore the level of representation that is most crucial for cognition. Accordingly, state-of-the-art sensory substitution techniques contribute significantly to the understanding of how the brain processes sensory information and also represents it with distinct qualia. This progressively advances cognitive theories with respect to multisensory perception and cross-modal cognition. Due to its versatility, sensory substitution phenomena also carry the applications of cognitive theories to other interdisciplinary research areas such as human-computer interactions (HCI). In HCI, cross- modal displays utilise sensory substitution techniques to augment users by enabling them to acquire sensory information via a sensory channel of different origin. The modular and flexible nature of cross-modal displays provide a supplementary framework that can appeal to a wider range of people whose physical and cognitive capabilities vary on a continuum. The present thesis focuses on the inclusive applications of sensory substitution techniques and cross- modal displays. Chapter I outlines the inclusive design mindset and proposes a case for applications of sensory substitution techniques for all of us. Chapter II and Chapter IV evaluates cross-modal displays in digital emotion communication and navigation applications respectively. Chapter III offers a methodology to study sensory substitution in a multisensory context. The present thesis evidences that perception with cross-modal displays utilises the capabilities of various senses. It further investigates the implication of this and suggests that cross-modal displays can benefit from multisensory combination. With multisensory combination, cross-modal displays with unisensory and multisensory modes can deliver complementary feedback. In this way, it is argued users can gain access to the same inclusive information technology with customised sensory channels. Overall, the scope of the present thesis approaches sensory substitution phenomena from an HCI perspective with theoretical implications grounded in cognitive sciences.

Date of Award	24 Jun 2020
Original language	English
Awarding Institution	University of Bath
Sponsors	Marie Skłodowska-Curie Fellowship for Industrial Research Enhancement
Supervisor	Eamonn O'Neill (Supervisor) & Michael Proulx (Supervisor)