A Novel Neural Network Architecture with Applications to 3D Animation and Interaction in Virtual Reality

  • Javier Dehesa

Student thesis: Doctoral ThesisDoctor of Engineering (EngD)


Realistic real-time character animation in 3D virtual environments is a difficult task, especially as graphics fidelity and production standards continue to rise in the industry. Motion capture is an essential tool to produce lifelike animation, but by itself it does not solve the complexities inherent to real-time environments. In the case of interactive characters in virtual reality, this difficulty is taken to another level, as the animation must dynamically adapt to the free actions of the user. On top of that, these actions, unlike the touch of a button, are not easily interpretable, introducing new challenges to interaction design.We propose a novel data-driven approach to these problems that takes most of this complexity out of the hands of the developers and into automated machine learning methods.

We propose a new neural network architecture, “grid-functioned neural networks” (GFNN), that is particularly well suited to model animation problems. Unlike previous proposals, GFNN features a grid of expert parameterisations associated with specific regions of a multidimensional domain, making it more capable of learning local patterns than conventional models. We give a full mathematical characterisation of this architecture as well as practical applications and results, evaluating its benefits as compared with other state-of-the-art models. We then propose a complete framework for human--character interaction in virtual reality built upon this model, along with gesture recognition and behaviour planning models. The framework establishes a novel general data-driven approach to the problem applicable to a variety of scenarios, as opposed to existing ad hoc solutions to specific cases. This is described at an abstract, general level and in the context of a particular case study, namely virtual sword fighting, demonstrating the practical implementation of these ideas.

Our results show that grid-functioned neural networks surpass other comparable models in aspects like control accuracy, predictability and computational performance, while the evaluation of our interaction framework case study situates it as a strong alternative to traditional animation and interaction development techniques. This contributes to the growing trend of incorporating data-driven systems into video games and other interactive media, which we foresee as a convergent future for industry and academia.
Date of Award16 Jun 2021
Original languageEnglish
Awarding Institution
  • University of Bath
SponsorsNinja Theory Ltd
SupervisorJulian Padget (Supervisor), Christof Lutteroth (Supervisor) & Andrew Vidler (Supervisor)


  • neural networks
  • machine learning
  • animation
  • human-computer interaction
  • virtual reality

Cite this