Intelligent video editing: Incorporating modern talking face generation algorithms in a video editor

Anchit Gupta, Faizan Farooq Khan, Rudrabha Mukhopadhyay, Vinay P. Namboodiri, C. V. Jawahar

Research output: Chapter or section in a book/report/conference proceedingChapter in a published conference proceeding

Abstract

This paper proposes a video editor based on OpenShot with several state-of-the-art facial video editing algorithms as added functionalities. Our editor provides an easy-to-use interface to apply modern lip-syncing algorithms interactively. Apart from lip-syncing, the editor also uses audio and facial re-enactment to generate expressive talking faces. The manual control improves the overall experience of video editing without missing out on the benefits of modern synthetic video generation algorithms. This control enables us to lip-sync complex dubbed movie scenes, interviews, television shows, and other visual content. Furthermore, our editor provides features that automatically translate lectures from spoken content, lip-sync of the professor, and background content like slides. While doing so, we also tackle the critical aspect of synchronizing background content with the translated speech. We qualitatively evaluate the usefulness of the proposed editor by conducting human evaluations. Our evaluations show a clear improvement in the efficiency of using human editors and an improved video generation quality. We attach demo videos with the supplementary material clearly explaining the tool and also showcasing multiple results.

Original languageEnglish
Title of host publicationProceedings of ICVGIP 2021 - 12th Indian Conference on Computer Vision, Graphics and Image Processing
Place of PublicationU. S. A.
PublisherAssociation for Computing Machinery
Pages1-9
ISBN (Electronic)9781450391276
DOIs
Publication statusPublished - 19 Dec 2021
Event12th Indian Conference on Computer Vision, Graphics and Image Processing, ICVGIP 2021 - Virtual, Online, India
Duration: 20 Dec 202122 Dec 2021

Publication series

NameACM International Conference Proceeding Series

Conference

Conference12th Indian Conference on Computer Vision, Graphics and Image Processing, ICVGIP 2021
Country/TerritoryIndia
CityVirtual, Online
Period20/12/2122/12/21

Keywords

  • Human in the loop
  • Lip-sync
  • Speech-to-speech translation
  • Talking head generation
  • Video editing

ASJC Scopus subject areas

  • Human-Computer Interaction
  • Computer Networks and Communications
  • Computer Vision and Pattern Recognition
  • Software

Fingerprint

Dive into the research topics of 'Intelligent video editing: Incorporating modern talking face generation algorithms in a video editor'. Together they form a unique fingerprint.

Cite this