Parametric Reshaping of Portraits in Videos

Xiangjun Tang, Wenxin Sun, Yongliang Yang, Xiaogang Jin

Research output: Chapter in Book/Report/Conference proceedingChapter in a published conference proceeding

114 Downloads (Pure)

Abstract

Sharing short personalized videos to various social media networks has become quite popular in recent years. This raises the need for digital retouching of portraits in videos. However, applying portrait image editing directly on portrait video frames cannot generate smooth and stable video sequences. To this end, we present a robust and easy-to-use parametric method to reshape the portrait in a video to produce smooth retouched results. Given an input portrait video, our method consists of two main stages: stabilized face reconstruction, and continuous video reshaping. In the first stage, we start by estimating face rigid pose transformations across video frames. Then we jointly optimize multiple frames to reconstruct an accurate face identity, followed by recovering face expressions over the entire video. In the second stage, we first reshape the reconstructed 3D face using a parametric reshaping model reflecting the weight change of the face, and then utilize the reshaped 3D face to guide the warping of video frames. We develop a novel signed distance function based dense mapping method for the warping between face contours before and after reshaping, resulting in stable warped video frames with minimum distortions. In addition, we use the 3D structure of the face to correct the dense mapping to achieve temporal consistency. We generate the final result by minimizing the background distortion through optimizing a content-aware warping mesh. Extensive experiments show that our method is able to create visually pleasing results by adjusting a simple reshaping parameter, which facilitates portrait video editing for social media and visual effects.
Original languageEnglish
Title of host publicationMM 2021 - Proceedings of the 29th ACM International Conference on Multimedia
Place of PublicationU. S. A.
PublisherAssociation for Computing Machinery
Pages4689-4697
Number of pages9
ISBN (Electronic)9781450386517
DOIs
Publication statusPublished - 17 Oct 2021
Event29th ACM International Conference on Multimedia, MM 2021 -
Duration: 20 Oct 202124 Oct 2021

Publication series

NameMM 2021 - Proceedings of the 29th ACM International Conference on Multimedia

Conference

Conference29th ACM International Conference on Multimedia, MM 2021
Period20/10/2124/10/21

Keywords

  • face reconstruction
  • face reshaping
  • video portrait editing

ASJC Scopus subject areas

  • Human-Computer Interaction
  • Software
  • Computer Graphics and Computer-Aided Design

Fingerprint

Dive into the research topics of 'Parametric Reshaping of Portraits in Videos'. Together they form a unique fingerprint.

Cite this