Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis

K. R. Prajwal, Rudrabha Mukhopadhyay, Vinay Namboodiri, C. V. Jawahar

Research output: Chapter or section in a book/report/conference proceedingChapter in a published conference proceeding

73 Citations (SciVal)
81 Downloads (Pure)


Humans involuntarily tend to infer parts of the conversation from lip movements when the speech is absent or corrupted by external noise. In this work, we explore the task of lip to speech synthesis, i.e., learning to generate natural speech given only the lip movements of a speaker. Acknowledging the importance of contextual and speaker-specific cues for accurate lip reading, we take a different path from existing works. We focus on learning accurate lip sequences to speech mappings for individual speakers in unconstrained, large vocabulary settings. To this end, we collect and release a large-scale benchmark dataset, the first of its kind, specifically to train and evaluate the single-speaker lip to speech task in natural settings. We propose an approach to achieve accurate, natural lip to speech synthesis in such unconstrained scenarios for the first time. Extensive evaluation using quantitative, qualitative metrics and human evaluation shows that our method is almost twice as intelligible as previous works in this space.
Original languageEnglish
Title of host publicationIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Place of PublicationSeattle, USA
Number of pages10
Publication statusPublished - 5 Aug 2020


Dive into the research topics of 'Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis'. Together they form a unique fingerprint.

Cite this