Free-Viewpoint Facial Re-Enactment from a Casual Capture

Srinivas Rao; Rodrigo Ortiz-Cayon; Matteo Munaro; Aidas Liaudanskas; Krunal Chande; Tobias Bertel; Christian Richardt; Alexander J. B. Trevor; Stefan Holzer; Abhishek Kar

doi:10.1145/3415264.3425453

Free-Viewpoint Facial Re-Enactment from a Casual Capture

Srinivas Rao
, Rodrigo Ortiz-Cayon
, Matteo Munaro
, Aidas Liaudanskas
, Krunal Chande
, Tobias Bertel
, Christian Richardt
, Alexander J. B. Trevor
, Stefan Holzer
, Abhishek Kar

Fyusion Inc.
Robust.AI
Google Inc

Research output: Chapter or section in a book/report/conference proceeding › Chapter in a published conference proceeding

4 Citations (SciVal)

368 Downloads (Pure)

Abstract

We propose a system for free-viewpoint facial re-enactment from a casual video capture of a target subject. Our system can render and re-enact the subject consistently in all the captured views. Furthermore, our system also enables interactive free-viewpoint facial re-enactment of the target from novel views. The re-enactment of the target subject is driven by an expression sequence of a source subject, which is captured using a custom app running on an iPhone X. Our system handles large pose variations in the target subject while keeping the re-enactment consistent. We demonstrate the efficacy of our system by showing various applications.

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 665992

Original language	English
Title of host publication	SA '20 Posters: SIGGRAPH Asia 2020 Posters
Publisher	Association for Computing Machinery
Pages	1-2
Number of pages	2
DOIs	https://doi.org/10.1145/3415264.3425453
Publication status	Published - 31 Dec 2020
Event	ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques in Asia - Online, Online, Korea, Republic of Duration: 4 Dec 2020 → 13 Dec 2020 Conference number: 13 https://sa2020.siggraph.org/en/

Conference

Conference	ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques in Asia
Abbreviated title	SIGGRAPH Asia
Country/Territory	Korea, Republic of
City	Online
Period	4/12/20 → 13/12/20
Internet address	https://sa2020.siggraph.org/en/

Keywords

facial reenactment
neural network
image-based rendering

Access to Document

10.1145/3415264.3425453

HoloBot-RaoEtAl-SA2020-Poster-Abstract.PDF
© ACM, 2020. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in SA '20 Posters: SIGGRAPH Asia 2020 Posters. http://doi.acm.org/10.1145/3415264.3425453
Accepted author manuscript, 4.42 MB

Centre for the Analysis of Motion, Entertainment Research and Applications (CAMERA) - 2.0
Campbell, N. (PI), Cosker, D. (PI), Bilzon, J. (CoI), Campbell, N. (CoI), Cazzola, D. (CoI), Colyer, S. (CoI), Cosker, D. (CoI), Lutteroth, C. (CoI), McGuigan, P. (CoI), O'Neill, E. (CoI), Petrini, K. (CoI), Proulx, M. (CoI) & Yang, Y. (CoI)
EPSRC
1/11/20 → 31/10/25
Project: Research council
Centre for the Analysis of Motion, Entertainment Research and Applications (CAMERA)
Cosker, D. (PI), Bilzon, J. (CoI), Campbell, N. (CoI), Cazzola, D. (CoI), Colyer, S. (CoI), Fincham Haines, T. (CoI), Hall, P. (CoI), Kim, K. I. (CoI), Lutteroth, C. (CoI), McGuigan, P. (CoI), O'Neill, E. (CoI), Richardt, C. (CoI), Salo, A. (CoI), Seminati, E. (CoI), Tabor, A. (CoI) & Yang, Y. (CoI)
EPSRC
1/09/15 → 28/02/21
Project: Research council

Cite this

Rao, S, Ortiz-Cayon, R, Munaro, M, Liaudanskas, A, Chande, K, Bertel, T, Richardt, C, Trevor, AJB, Holzer, S & Kar, A 2020, Free-Viewpoint Facial Re-Enactment from a Casual Capture. in SA '20 Posters: SIGGRAPH Asia 2020 Posters., 37, Association for Computing Machinery, pp. 1-2, ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques in Asia, Online, Korea, Republic of, 4/12/20. https://doi.org/10.1145/3415264.3425453

@inproceedings{6b200e4923ab40749c975d71d9da180e,

title = "Free-Viewpoint Facial Re-Enactment from a Casual Capture",

abstract = "We propose a system for free-viewpoint facial re-enactment from a casual video capture of a target subject. Our system can render and re-enact the subject consistently in all the captured views. Furthermore, our system also enables interactive free-viewpoint facial re-enactment of the target from novel views. The re-enactment of the target subject is driven by an expression sequence of a source subject, which is captured using a custom app running on an iPhone X. Our system handles large pose variations in the target subject while keeping the re-enactment consistent. We demonstrate the efficacy of our system by showing various applications. This project has received funding from the European Union{\textquoteright}s Horizon 2020 research and innovation programme under the Marie Sk{\l}odowska-Curie grant agreement No 665992 ",

keywords = "facial reenactment, neural network, image-based rendering",

author = "Srinivas Rao and Rodrigo Ortiz-Cayon and Matteo Munaro and Aidas Liaudanskas and Krunal Chande and Tobias Bertel and Christian Richardt and Trevor, \{Alexander J. B.\} and Stefan Holzer and Abhishek Kar",

year = "2020",

month = dec,

day = "31",

doi = "10.1145/3415264.3425453",

language = "English",

pages = "1--2",

booktitle = "SA '20 Posters: SIGGRAPH Asia 2020 Posters",

publisher = "Association for Computing Machinery",

address = "USA United States",

note = "ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques in Asia, SIGGRAPH Asia ; Conference date: 04-12-2020 Through 13-12-2020",

url = "https://sa2020.siggraph.org/en/",

}

TY - GEN

T1 - Free-Viewpoint Facial Re-Enactment from a Casual Capture

AU - Rao, Srinivas

AU - Ortiz-Cayon, Rodrigo

AU - Munaro, Matteo

AU - Liaudanskas, Aidas

AU - Chande, Krunal

AU - Bertel, Tobias

AU - Richardt, Christian

AU - Trevor, Alexander J. B.

AU - Holzer, Stefan

AU - Kar, Abhishek

N1 - Conference code: 13

PY - 2020/12/31

Y1 - 2020/12/31

N2 - We propose a system for free-viewpoint facial re-enactment from a casual video capture of a target subject. Our system can render and re-enact the subject consistently in all the captured views. Furthermore, our system also enables interactive free-viewpoint facial re-enactment of the target from novel views. The re-enactment of the target subject is driven by an expression sequence of a source subject, which is captured using a custom app running on an iPhone X. Our system handles large pose variations in the target subject while keeping the re-enactment consistent. We demonstrate the efficacy of our system by showing various applications. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 665992

AB - We propose a system for free-viewpoint facial re-enactment from a casual video capture of a target subject. Our system can render and re-enact the subject consistently in all the captured views. Furthermore, our system also enables interactive free-viewpoint facial re-enactment of the target from novel views. The re-enactment of the target subject is driven by an expression sequence of a source subject, which is captured using a custom app running on an iPhone X. Our system handles large pose variations in the target subject while keeping the re-enactment consistent. We demonstrate the efficacy of our system by showing various applications. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 665992

KW - facial reenactment

KW - neural network

KW - image-based rendering

U2 - 10.1145/3415264.3425453

DO - 10.1145/3415264.3425453

M3 - Chapter in a published conference proceeding

SP - 1

EP - 2

BT - SA '20 Posters: SIGGRAPH Asia 2020 Posters

PB - Association for Computing Machinery

T2 - ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques in Asia

Y2 - 4 December 2020 through 13 December 2020

ER -

Free-Viewpoint Facial Re-Enactment from a Casual Capture

Abstract

Conference

Keywords

Access to Document

Fingerprint

Projects

Centre for the Analysis of Motion, Entertainment Research and Applications (CAMERA) - 2.0

Centre for the Analysis of Motion, Entertainment Research and Applications (CAMERA)

Cite this