Live Intrinsic Video

Abhimitra Meka, Michael Zollhöfer, Christian Richardt, Christian Theobalt

Research output: Contribution to journalArticlepeer-review

66 Citations (SciVal)
140 Downloads (Pure)


Intrinsic video decomposition refers to the fundamentally ambiguous task of separating a video stream into its constituent layers, in particular reflectance and shading layers. Such a decomposition is the basis for a variety of video manipulation applications, such as realistic recoloring or retexturing of objects. We present a novel variational approach to tackle this underconstrained inverse problem at real-time frame rates, which enables on-line processing of live video footage. The problem of finding the intrinsic decomposition is formulated as a mixed variational l2–lp optimization problem based on an objective function that is specifically tailored for fast optimization. To this end, we propose a novel combination of sophisticated local spatial and global spatio-temporal priors resulting in temporally coherent decompositions at real-time frame rates without the need for explicit correspondence search. We tackle the resulting high-dimensional, non-convex optimization problem via a novel data-parallel iteratively reweighted least squares solver that runs on commodity graphics hardware. Real-time performance is obtained by combining a local-global solution strategy with hierarchical coarse-to-fine optimization. Compelling real-time augmented reality applications, such as recoloring, material editing and retexturing, are demonstrated in a live setup. Our qualitative and quantitative evaluation shows that we obtain high-quality real-time decompositions even for challenging sequences. Our method is able to outperform state-of-the-art approaches in terms of runtime and result quality – even without user guidance such as scribbles.
Original languageEnglish
Article number109
Pages (from-to)1-14
Number of pages14
JournalACM Transactions on Graphics
Issue number4
Early online date1 Jul 2016
Publication statusPublished - 31 Jul 2016


  • intrinsic decomposition
  • reflectance
  • shading
  • p-norm
  • real time
  • data-parallel optimization
  • recoloring
  • retexturing


Dive into the research topics of 'Live Intrinsic Video'. Together they form a unique fingerprint.

Cite this