Reactive Video: Movement Sonification as Auditory Feedback for Supporting Physical Activity

Doga Cavdir, Christopher Clarke, Patrick Chiu, Laurent Denoue, Don Kimber

Research output: Contribution to conferencePaperpeer-review

63 Downloads (Pure)


This paper provides initial efforts in developing and evaluating a real-time movement sonification framework for physical activity practice and learning. Reactive Video provides an interactive, vision-based, adaptive video playback with auditory feedback on users' performance to better support when learning and practicing new physical skills. We implement the sonification for auditory feedback design by extending the Web Audio API framework. The current application focuses on Tai-Chi performance and provides two main audio cues to users for several Tai Chi exercises. We provide our design approach, implementation, and sound generation and mapping, specifically for interactive systems with direct video manipulation. Our observations reveal the relationship between the movement-to-sound mapping and characteristics of the physical activity.
Original languageEnglish
Number of pages11
Publication statusPublished - 14 Jun 2021
EventNew Interfaces for Musical Expression 2021 - Online, Shanghai, China
Duration: 14 Jun 202118 Jun 2021


ConferenceNew Interfaces for Musical Expression 2021
Abbreviated titleNIME 2021
Internet address


Dive into the research topics of 'Reactive Video: Movement Sonification as Auditory Feedback for Supporting Physical Activity'. Together they form a unique fingerprint.

Cite this