Real-time Global Illumination Decomposition of Videos

Abhimitra Meka, Mohammad Shafiei, Michael Zollhöfer, Christian Richardt, Christian Theobalt

Research output: Contribution to journalArticlepeer-review

5 Citations (SciVal)
57 Downloads (Pure)


We propose the first approach for the decomposition of a monocular color video into direct and indirect illumination components in real time. We retrieve, in separate layers, the contribution made to the scene appearance by the scene reflectance, the light sources and the reflections from various coherent scene regions to one another. Existing techniques that invert global light transport require image capture under multiplexed controlled lighting, or only enable the decomposition of a single image at slow off-line frame rates. In contrast, our approach works for regular videos and produces temporally coherent decomposition layers at real-time frame rates. At the core of our approach are several sparsity priors that enable the estimation of the per-pixel direct and indirect illumination layers based on a small set of jointly estimated base reflectance colors. The resulting variational decomposition problem uses a new formulation based on sparse and dense sets of non-linear equations that we solve efficiently using a novel alternating data-parallel optimization strategy. We evaluate our approach qualitatively and quantitatively, and show improvements over the state of the art in this field, in both quality and runtime. In addition, we demonstrate various realtime appearance editing applications for videos with consistent illumination.
Original languageEnglish
Article number22
Pages (from-to)1-16
Number of pages16
JournalACM Transactions on Graphics
Issue number3
Publication statusPublished - 30 Jun 2021


  • Illumination decomposition
  • direct and indirect illumination
  • real-time sparse-dense optimization

ASJC Scopus subject areas

  • Computer Graphics and Computer-Aided Design


Dive into the research topics of 'Real-time Global Illumination Decomposition of Videos'. Together they form a unique fingerprint.

Cite this