A Shared Control Strategy for Reach and Grasp of Multiple Objects Using Robot Vision and Noninvasive Brain-Computer Interface

Yang Xu, Heng Zhang, Linfeng Cao, Xiaokang Shu, Dingguo Zhang

Research output: Contribution to journalArticlepeer-review

17 Citations (SciVal)


It is ambitious to develop a brain-controlled robotic arm for some patients with motor impairments to perform activities of daily living using brain-computer interfaces (BCIs). Despite much progress achieved, this mission is still very challenging mainly due to the poor decoding performance of BCIs. The problem is even exacerbated in the case of noninvasive BCIs. A shared control strategy is developed in this work to realize flexible robotic arm control for reach and grasp of multiple objects. With the intelligent assistance provided by robot vision, the subject was only required to finish gross reaching movement and target selection using a simple motor imagery-based BCI with binary output. Along with the user control, the robotic arm, which identified and localized potential targets within the workspace in the background, was capable of providing both trajectory correction in the reaching phase to reduce trajectory redundancy and autonomous grasping assistance in the phase of grasp. Ten subjects participated in the experiments containing one session of two-block grasping tasks with fixed locations and another one of randomly placed three-block grasping tasks. The results of the experiments demonstrated substantial improvement with the shared control system. Compared with the single BCI control, the success rate of shared control was significantly higher ( $p < 0.001$ for group performance), and moreover, the task completion time and perceived difficulty were significantly lower ( $p < 0.001$ for group performance both), indicating the potential of our proposed shared control system in real applications. Note to Practitioners - This article is motivated by the problem of dexterous robotic arm control based on a brain-computer interface (BCI). For people suffering from severe neuromuscular disorders or accident injuries, a brain-controlled robotic arm is expected to provide assistance in their daily lives. A primary bottleneck to achieve the objective is that the information transfer rate of current BCIs is not high enough to produce multiple and reliable commands during the online robotic control. In this work, machine autonomy is incorporated in a BCI-controlled robotic arm system, where the user and machine can work together to reach and grasp multiple objects in a given task. The intelligent robot system autonomously localized the potential targets and provided trajectory correction and grasping assistance accordingly. Meanwhile, the user only needed to complete gross reaching movement and target selection with a basic binary motor imagery-based BCI, which reduced the task difficulty and retained the volitional involvement of the user at the same time. The results of the experiments showed that the accuracy and efficiency of grasping tasks increased significantly in the shared control mode together with a significant decrease in the perceived mental workload, which indicates that our proposed shared control system is effective and user-friendly in practice. In the future, more feedback information will be introduced to enhance the task performance further, and a wheelchair-mounted robotic arm system will be developed for greater flexibility. In addition, more functional task modules (e.g., self-feeding and opening doors) should be integrated for more practical utilities.

Original languageEnglish
Pages (from-to)360-372
Number of pages13
JournalIEEE Transactions on Automation Science and Engineering
Issue number1
Early online date16 Nov 2020
Publication statusPublished - 31 Jan 2022

Bibliographical note

Funding Information:
This work was supported in part by the National Natural Science Foundation of China under Grant 61761166006 and Grant 91848112, in part by the National Key Research and Development Program of China under Grant 2018YFB1307301, and in part by the Shanghai Municipal Commission of Health and Family Planning under Grant 2017ZZ01006.

Publisher Copyright:
© 2004-2012 IEEE.


  • Brain-computer interface (BCI)
  • motor imagery (MI)
  • robot vision
  • robotic arm
  • shared control

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Electrical and Electronic Engineering


Dive into the research topics of 'A Shared Control Strategy for Reach and Grasp of Multiple Objects Using Robot Vision and Noninvasive Brain-Computer Interface'. Together they form a unique fingerprint.

Cite this