Towards automatic face-to-face translation

K. R. Prajwal, Rudrabha Mukhopadhyay, Jerin Philip, Abhishek Jha, Vinay Namboodiri, C. V. Jawahar

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

In light of the recent breakthroughs in automatic machine translation systems, we propose a novel approach that we term as "Face-to-Face Translation". As today's digital communication becomes increasingly visual, we argue that there is a need for systems that can automatically translate a video of a person speaking in language A into a target language B with realistic lip synchronization. In this work, we create an automatic pipeline for this problem and demonstrate its impact in multiple real-world applications. First, we build a working speech-to-speech translation system by bringing together multiple existing modules from speech and language. We then move towards "Face-to-Face Translation" by incorporating a novel visual module, LipGAN for generating realistic talking faces from the translated audio. Quantitative evaluation of LipGAN on the standard LRW test set shows that it significantly outperforms existing approaches across all standard metrics. We also subject our Face-to-Face Translation pipeline, to multiple human evaluations and show that it can significantly improve the overall user experience for consuming and interacting with multimodal content across languages. Code, models and demo video are made publicly available.

Original languageEnglish
Title of host publicationMM 2019 - Proceedings of the 27th ACM International Conference on Multimedia
PublisherAssociation for Computing Machinery
Pages1428-1436
Number of pages9
ISBN (Electronic)9781450368896
DOIs
Publication statusPublished - 15 Oct 2019
Event27th ACM International Conference on Multimedia, MM 2019 - Nice, France
Duration: 21 Oct 201925 Oct 2019

Publication series

NameMM 2019 - Proceedings of the 27th ACM International Conference on Multimedia

Conference

Conference27th ACM International Conference on Multimedia, MM 2019
CountryFrance
CityNice
Period21/10/1925/10/19

Keywords

  • Cross-language talking face generation
  • Lip Synthesis
  • Neural Machine Translation
  • Speech to Speech Translation
  • Translation systems
  • Voice Transfer

ASJC Scopus subject areas

  • Media Technology
  • Computer Science(all)

Cite this

Prajwal, K. R., Mukhopadhyay, R., Philip, J., Jha, A., Namboodiri, V., & Jawahar, C. V. (2019). Towards automatic face-to-face translation. In MM 2019 - Proceedings of the 27th ACM International Conference on Multimedia (pp. 1428-1436). (MM 2019 - Proceedings of the 27th ACM International Conference on Multimedia). Association for Computing Machinery. https://doi.org/10.1145/3343031.3351066