Collaborative architecture for human-robot assembly tasks using multimodal sensors

Human robot collaboration in manufacturing environments lacks adaptability to different tasks and environments as well as fluency of interaction with workers. This work looks at developing a cognitive architecture for assembly robots allowing prediction of when collaborative actions are required. The cognitive architecture provides reliable perception and reasoning methods for increased human-robot fluency allowing the user to focus on the required task. The system has three layers; the perception layer determines the current task state while the memory layer keeps track of task details, current predictions and past episodes. The control layer predicts future collaborative actions and passes commands to the robot at the required time to reduce user idle time. The system uses inertial measurement unit convolutional neural network action recognition and vision recognition for environment perception. The cognitive architecture is validated with experiments in offline and real-time modes. In offline mode, two action recognition methods are performed, with one classifier for all actions achieving accuracy of 81% and a separate classifier for each action achieving 74%. In real-time mode, a UR3 cobot is used for a collaborative assembly task where an increase in the proportion of time the user is active is shown.