Reactive Video: Movement Sonification as Auditory Feedback for Supporting Physical Activity

Doga Cavdir, Christopher Clarke, Patrick Chiu, Laurent Denoue, Don Kimber

Research output: Chapter in Book/Report/Conference proceedingConference contribution

4 Downloads (Pure)


This paper provides initial efforts in developing and evaluating a real-time movement sonification framework for physical activity practice and learning. Reactive Video provides an interactive, vision-based, adaptive video playback with auditory feedback on users' performance to better support when learning and practicing new physical skills. We implement the sonification for auditory feedback design by extending the Web Audio API framework. The current application focuses on Tai-Chi performance and provides two main audio cues to users for several Tai Chi exercises. We provide our design approach, implementation, and sound generation and mapping, specifically for interactive systems with direct video manipulation. Our observations reveal the relationship between the movement-to-sound mapping and characteristics of the physical activity.
Original languageEnglish
Title of host publicationNew Interfaces for Musical Expression (NIME) 2021
Publication statusAcceptance date - 22 Apr 2021

Cite this