Photorealistic Audio-driven Video Portraits

Xin Wen, Miao Wang, Christian Richardt, Ze-Yin Chen, Shi-Min Hi

Research output: Contribution to journalArticlepeer-review

279 Downloads (Pure)

Abstract

Video portraits are common in a variety of applications, such as videoconferencing, news broadcasting, and virtual education and training. We present a novel method to synthesize photorealistic video portraits for an input portrait video, automatically driven by a person’s voice. The main challenge in this task is the hallucination of plausible, photorealistic facial expressions from input speech audio. To address this challenge, we employ a parametric 3D face model represented by geometry, facial expression, illumination, etc., and learn a mapping from audio features to model parameters. The input source audio is first represented as a high-dimensional feature, which is used to predict facial expression parameters of the 3D face model. We then replace the expression parameters computed from the original target video with the predicted one, and rerender the reenacted face. Finally, we generate a photorealistic video portrait from the reenacted synthetic face sequence via a neural face renderer. One appealing feature of our approach is the generalization capability for various input speech audio, including synthetic speech audio from text-to-speech software. Extensive experimental results show that our approach outperforms previous general-purpose audio-driven video portrait methods. This includes a user study demonstrating that our results are rated as more realistic than previous methods.
Original languageEnglish
Pages (from-to)3457-3466
Number of pages10
JournalIEEE Transactions on Visualization and Computer Graphics
Volume26
Issue number12
Early online date17 Sep 2020
DOIs
Publication statusPublished - 31 Dec 2020
EventInternational Symposium on Mixed and Augmented Reality - Online
Duration: 9 Nov 202013 Nov 2020
http://ismar20.org/

Keywords

  • audio-driven animation
  • facial reenactment
  • generative models
  • talking-head video generation

Fingerprint

Dive into the research topics of 'Photorealistic Audio-driven Video Portraits'. Together they form a unique fingerprint.

Cite this