EchoFlex: Hand gesture recognition using ultrasound imaging

Jess McIntosh; Asier Marzo; Mike Fraser; Carol Phillips

doi:10.1145/3025453.3025807

EchoFlex: Hand gesture recognition using ultrasound imaging

Jess McIntosh, Asier Marzo, Mike Fraser, Carol Phillips

University of Bristol

Research output: Chapter or section in a book/report/conference proceeding › Chapter in a published conference proceeding

87 Citations (SciVal)

Abstract

Recent improvements in ultrasound imaging enable new opportunities for hand pose detection using wearable devices. Ultrasound imaging has remained under-explored in the HCI community despite being non-invasive, harmless and capable of imaging internal body parts, with applications including smart-watch interaction, prosthesis control and instrument tuition. In this paper, we compare the performance of different forearm mounting positions for a wearable ultrasonographic device. Location plays a fundamental role in ergonomics and performance since the anatomical features differ among positions. We also investigate the performance decrease due to cross-session position shifts and develop a technique to compensate for this misalignment. Our gesture recognition algorithm combines image processing and neural networks to classify the flexion and extension of 10 discrete hand gestures with an accuracy above 98%. Furthermore, this approach can continuously track individual digit flexion with less than 5% NRMSE, and also differentiate between digit flexion at different joints.

Original language	English
Title of host publication	CHI '17: Proceedings of the 2017 ACM SIGCHI Conference on Human Factors in Computing Systems
Subtitle of host publication	Explore, Innovate, Inspire
Publisher	Association for Computing Machinery
Pages	1923-1934
Number of pages	12
ISBN (Electronic)	9781450346559
DOIs	https://doi.org/10.1145/3025453.3025807
Publication status	Published - 2 May 2017
Event	2017 ACM SIGCHI Conference on Human Factors in Computing Systems, CHI 2017 - Denver, USA United States Duration: 6 May 2017 → 11 May 2017

Publication series

Name	Conference on Human Factors in Computing Systems - Proceedings
Volume	2017-May

Conference

Conference	2017 ACM SIGCHI Conference on Human Factors in Computing Systems, CHI 2017
Country/Territory	USA United States
City	Denver
Period	6/05/17 → 11/05/17

Funding

This research was supported by EPSRC Doctoral Training funding through grant EP/M507994/1. We especially thank members of BIRCH within the UH Bristol NHS Foundation Trust. We also thank Nvidia for facilitating our research by providing hardware and Matthew Sutton for assistance with the video and pictures.

Keywords

Computer vision
Gesture recognition
Interactive ultrasound imaging
Machine learning

ASJC Scopus subject areas

Software
Human-Computer Interaction
Computer Graphics and Computer-Aided Design

Access to Document

10.1145/3025453.3025807

Cite this

McIntosh, J., Marzo, A., Fraser, M., & Phillips, C. (2017). EchoFlex: Hand gesture recognition using ultrasound imaging. In CHI '17: Proceedings of the 2017 ACM SIGCHI Conference on Human Factors in Computing Systems: Explore, Innovate, Inspire (pp. 1923-1934). (Conference on Human Factors in Computing Systems - Proceedings; Vol. 2017-May). Association for Computing Machinery. https://doi.org/10.1145/3025453.3025807

EchoFlex: Hand gesture recognition using ultrasound imaging. / McIntosh, Jess; Marzo, Asier; Fraser, Mike et al.
CHI '17: Proceedings of the 2017 ACM SIGCHI Conference on Human Factors in Computing Systems: Explore, Innovate, Inspire. Association for Computing Machinery, 2017. p. 1923-1934 (Conference on Human Factors in Computing Systems - Proceedings; Vol. 2017-May).

Research output: Chapter or section in a book/report/conference proceeding › Chapter in a published conference proceeding

McIntosh, J, Marzo, A, Fraser, M & Phillips, C 2017, EchoFlex: Hand gesture recognition using ultrasound imaging. in CHI '17: Proceedings of the 2017 ACM SIGCHI Conference on Human Factors in Computing Systems: Explore, Innovate, Inspire. Conference on Human Factors in Computing Systems - Proceedings, vol. 2017-May, Association for Computing Machinery, pp. 1923-1934, 2017 ACM SIGCHI Conference on Human Factors in Computing Systems, CHI 2017, Denver, USA United States, 6/05/17. https://doi.org/10.1145/3025453.3025807

McIntosh J, Marzo A, Fraser M, Phillips C. EchoFlex: Hand gesture recognition using ultrasound imaging. In CHI '17: Proceedings of the 2017 ACM SIGCHI Conference on Human Factors in Computing Systems: Explore, Innovate, Inspire. Association for Computing Machinery. 2017. p. 1923-1934. (Conference on Human Factors in Computing Systems - Proceedings). doi: 10.1145/3025453.3025807

@inproceedings{1d2f1a2dd0ef4cf99fda820ecf913cfb,

title = "EchoFlex: Hand gesture recognition using ultrasound imaging",

abstract = "Recent improvements in ultrasound imaging enable new opportunities for hand pose detection using wearable devices. Ultrasound imaging has remained under-explored in the HCI community despite being non-invasive, harmless and capable of imaging internal body parts, with applications including smart-watch interaction, prosthesis control and instrument tuition. In this paper, we compare the performance of different forearm mounting positions for a wearable ultrasonographic device. Location plays a fundamental role in ergonomics and performance since the anatomical features differ among positions. We also investigate the performance decrease due to cross-session position shifts and develop a technique to compensate for this misalignment. Our gesture recognition algorithm combines image processing and neural networks to classify the flexion and extension of 10 discrete hand gestures with an accuracy above 98%. Furthermore, this approach can continuously track individual digit flexion with less than 5% NRMSE, and also differentiate between digit flexion at different joints.",

keywords = "Computer vision, Gesture recognition, Interactive ultrasound imaging, Machine learning",

author = "Jess McIntosh and Asier Marzo and Mike Fraser and Carol Phillips",

year = "2017",

month = may,

day = "2",

doi = "10.1145/3025453.3025807",

language = "English",

series = "Conference on Human Factors in Computing Systems - Proceedings",

publisher = "Association for Computing Machinery",

pages = "1923--1934",

booktitle = "CHI '17: Proceedings of the 2017 ACM SIGCHI Conference on Human Factors in Computing Systems",

address = "USA United States",

note = "2017 ACM SIGCHI Conference on Human Factors in Computing Systems, CHI 2017 ; Conference date: 06-05-2017 Through 11-05-2017",

}

TY - GEN

T1 - EchoFlex

T2 - 2017 ACM SIGCHI Conference on Human Factors in Computing Systems, CHI 2017

AU - McIntosh, Jess

AU - Marzo, Asier

AU - Fraser, Mike

AU - Phillips, Carol

PY - 2017/5/2

Y1 - 2017/5/2

N2 - Recent improvements in ultrasound imaging enable new opportunities for hand pose detection using wearable devices. Ultrasound imaging has remained under-explored in the HCI community despite being non-invasive, harmless and capable of imaging internal body parts, with applications including smart-watch interaction, prosthesis control and instrument tuition. In this paper, we compare the performance of different forearm mounting positions for a wearable ultrasonographic device. Location plays a fundamental role in ergonomics and performance since the anatomical features differ among positions. We also investigate the performance decrease due to cross-session position shifts and develop a technique to compensate for this misalignment. Our gesture recognition algorithm combines image processing and neural networks to classify the flexion and extension of 10 discrete hand gestures with an accuracy above 98%. Furthermore, this approach can continuously track individual digit flexion with less than 5% NRMSE, and also differentiate between digit flexion at different joints.

AB - Recent improvements in ultrasound imaging enable new opportunities for hand pose detection using wearable devices. Ultrasound imaging has remained under-explored in the HCI community despite being non-invasive, harmless and capable of imaging internal body parts, with applications including smart-watch interaction, prosthesis control and instrument tuition. In this paper, we compare the performance of different forearm mounting positions for a wearable ultrasonographic device. Location plays a fundamental role in ergonomics and performance since the anatomical features differ among positions. We also investigate the performance decrease due to cross-session position shifts and develop a technique to compensate for this misalignment. Our gesture recognition algorithm combines image processing and neural networks to classify the flexion and extension of 10 discrete hand gestures with an accuracy above 98%. Furthermore, this approach can continuously track individual digit flexion with less than 5% NRMSE, and also differentiate between digit flexion at different joints.

KW - Computer vision

KW - Gesture recognition

KW - Interactive ultrasound imaging

KW - Machine learning

UR - http://www.scopus.com/inward/record.url?scp=85034737655&partnerID=8YFLogxK

U2 - 10.1145/3025453.3025807

DO - 10.1145/3025453.3025807

M3 - Chapter in a published conference proceeding

AN - SCOPUS:85034737655

T3 - Conference on Human Factors in Computing Systems - Proceedings

SP - 1923

EP - 1934

BT - CHI '17: Proceedings of the 2017 ACM SIGCHI Conference on Human Factors in Computing Systems

PB - Association for Computing Machinery

Y2 - 6 May 2017 through 11 May 2017

ER -

EchoFlex: Hand gesture recognition using ultrasound imaging

Abstract

Publication series

Conference

Funding

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this