Projects per year
Abstract
Video portraits are common in a variety of applications, such as videoconferencing, news broadcasting, and virtual education and training. We present a novel method to synthesize photorealistic video portraits for an input portrait video, automatically driven by a person’s voice. The main challenge in this task is the hallucination of plausible, photorealistic facial expressions from input speech audio. To address this challenge, we employ a parametric 3D face model represented by geometry, facial expression, illumination, etc., and learn a mapping from audio features to model parameters. The input source audio is first represented as a high-dimensional feature, which is used to predict facial expression parameters of the 3D face model. We then replace the expression parameters computed from the original target video with the predicted one, and rerender the reenacted face. Finally, we generate a photorealistic video portrait from the reenacted synthetic face sequence via a neural face renderer. One appealing feature of our approach is the generalization capability for various input speech audio, including synthetic speech audio from text-to-speech software. Extensive experimental results show that our approach outperforms previous general-purpose audio-driven video portrait methods. This includes a user study demonstrating that our results are rated as more realistic than previous methods.
Original language | English |
---|---|
Pages (from-to) | 3457-3466 |
Number of pages | 10 |
Journal | IEEE Transactions on Visualization and Computer Graphics |
Volume | 26 |
Issue number | 12 |
Early online date | 17 Sep 2020 |
DOIs | |
Publication status | Published - 31 Dec 2020 |
Event | International Symposium on Mixed and Augmented Reality - Online Duration: 9 Nov 2020 → 13 Nov 2020 http://ismar20.org/ |
Keywords
- audio-driven animation
- facial reenactment
- generative models
- talking-head video generation
Fingerprint Dive into the research topics of 'Photorealistic Audio-driven Video Portraits'. Together they form a unique fingerprint.
Projects
- 2 Active
-
Fellowship - Towards Immersive 360° VR Video with Motion Parallax
Engineering and Physical Sciences Research Council
25/06/18 → 24/06/21
Project: Research council
-
Centre for the Analysis of Motion, Entertainment Research and Applications (CAMERA)
Cosker, D., Campbell, N., Fincham Haines, T., Hall, P., Kim, K. I., Lutteroth, C., O'Neill, E., Richardt, C. & Yang, Y.
Engineering and Physical Sciences Research Council
1/09/15 → 28/02/21
Project: Research council