Photorealistic Audio-driven Video Portraits

Xin Wen, Miao Wang, Christian Richardt, Ze-Yin Chen, Shi-Min Hi

Research output: Contribution to journalArticlepeer-review

28 Citations (SciVal)
891 Downloads (Pure)


Video portraits are common in a variety of applications, such as videoconferencing, news broadcasting, and virtual education and training. We present a novel method to synthesize photorealistic video portraits for an input portrait video, automatically driven by a person’s voice. The main challenge in this task is the hallucination of plausible, photorealistic facial expressions from input speech audio. To address this challenge, we employ a parametric 3D face model represented by geometry, facial expression, illumination, etc., and learn a mapping from audio features to model parameters. The input source audio is first represented as a high-dimensional feature, which is used to predict facial expression parameters of the 3D face model. We then replace the expression parameters computed from the original target video with the predicted one, and rerender the reenacted face. Finally, we generate a photorealistic video portrait from the reenacted synthetic face sequence via a neural face renderer. One appealing feature of our approach is the generalization capability for various input speech audio, including synthetic speech audio from text-to-speech software. Extensive experimental results show that our approach outperforms previous general-purpose audio-driven video portrait methods. This includes a user study demonstrating that our results are rated as more realistic than previous methods.
Original languageEnglish
Pages (from-to)3457-3466
Number of pages10
JournalIEEE Transactions on Visualization and Computer Graphics
Issue number12
Early online date17 Sept 2020
Publication statusPublished - 31 Dec 2020
EventInternational Symposium on Mixed and Augmented Reality - Online
Duration: 9 Nov 202013 Nov 2020


  • audio-driven animation
  • facial reenactment
  • generative models
  • talking-head video generation


Dive into the research topics of 'Photorealistic Audio-driven Video Portraits'. Together they form a unique fingerprint.

Cite this