Abstract
Improvements in both software and hardware, as well as an increase in consumer suitable equipment, have resulted in great advances in the fields of virtual reality (VR) and augmented reality (AR). A primary focus of immersive research, using VR or AR, is bridging the gap between real and virtual. The feeling of disconnect between worlds largely arises due to the means of interaction with the virtual environment and computer generated (CG) objects in the scene. While current interaction mechanisms (e.g. controllers or hand gestures) have improved greatly in recent years, there are still limitations which must be overcome to reach the full potential of interaction within immersive experiences. Thus, to create immersive VR and AR applications and training environments, an appropriate method for allowing participants to interact with the virtual environments and elements of that scene must be considered. There does not currently exist a platform to bring physical objects into virtual worlds without additional peripherals or the use of expensive motion capture setups and so to overcome this we need a real-time solution for capturing the behaviour of physical objects in order to animate CG representations in VR or add effects to real-world objects in AR.In this work, we consider different approaches for transporting physical objects into virtual and augmented environments and collaborate with Marshmallow Laser Feast to facilitate novel and engaging interactions within their immersive experiences. To do so, we design an end-to-end pipeline for creating interactive VR Props from physical objects, with focus on non-rigid objects with large, distinct deformations such as bends and folds. In this pipeline, the behaviour of the objects are predicted using deep neural networks (DNNs). Our networks predict model parameters and use these to animate virtual representations of objects in VR and AR applications. We experiment with 3 different DNNs (a standard ResNet34 and our custom VRProp-Net and VRProp Net+) and compare the outputs of each of these. We present both a fixed camera solution as well as an egocentric solution which predicts the shape and pose of objects in a moving first person view, allowing a flexible capture volume and offering more freedom in immersive experiences. Finally, motivated by the potential applications for hand-object tracking within mixed reality experiences, we design a novel dataset EgoInteraction. This is the first large scale dataset containing egocentric hand-object interaction sequences with 3D ground truth data for a range of rigid, articulated and non-rigid objects.
| Date of Award | 11 Oct 2021 |
|---|---|
| Original language | English |
| Awarding Institution |
|
| Supervisor | Darren Cosker (Supervisor), Neill Campbell (Supervisor) & Robin McNicholas (Supervisor) |