Live Intrinsic Video

Abhimitra Meka, Michael Zollhöfer, Christian Richardt, Christian Theobalt

Research output: Contribution to journalArticle

36 Citations (Scopus)
82 Downloads (Pure)

Abstract

Intrinsic video decomposition refers to the fundamentally ambiguous task of separating a video stream into its constituent layers, in particular reflectance and shading layers. Such a decomposition is the basis for a variety of video manipulation applications, such as realistic recoloring or retexturing of objects. We present a novel variational approach to tackle this underconstrained inverse problem at real-time frame rates, which enables on-line processing of live video footage. The problem of finding the intrinsic decomposition is formulated as a mixed variational l2–lp optimization problem based on an objective function that is specifically tailored for fast optimization. To this end, we propose a novel combination of sophisticated local spatial and global spatio-temporal priors resulting in temporally coherent decompositions at real-time frame rates without the need for explicit correspondence search. We tackle the resulting high-dimensional, non-convex optimization problem via a novel data-parallel iteratively reweighted least squares solver that runs on commodity graphics hardware. Real-time performance is obtained by combining a local-global solution strategy with hierarchical coarse-to-fine optimization. Compelling real-time augmented reality applications, such as recoloring, material editing and retexturing, are demonstrated in a live setup. Our qualitative and quantitative evaluation shows that we obtain high-quality real-time decompositions even for challenging sequences. Our method is able to outperform state-of-the-art approaches in terms of runtime and result quality – even without user guidance such as scribbles.
Original languageEnglish
Article number109
Pages (from-to)1-14
Number of pages14
JournalACM Transactions on Graphics
Volume35
Issue number4
Early online date1 Jul 2016
DOIs
Publication statusPublished - 31 Jul 2016

Keywords

  • intrinsic decomposition
  • reflectance
  • shading
  • p-norm
  • real time
  • data-parallel optimization
  • recoloring
  • retexturing

Fingerprint Dive into the research topics of 'Live Intrinsic Video'. Together they form a unique fingerprint.

  • Cite this

    Meka, A., Zollhöfer, M., Richardt, C., & Theobalt, C. (2016). Live Intrinsic Video. ACM Transactions on Graphics, 35(4), 1-14. [109]. https://doi.org/10.1145/2897824.2925907