Dynamic Mixed-Reality Compositing with Unity

We present a system for dynamic mixed-reality compositing, or how to insert dynamic computer-generated (CG) elements into live-action video footage in real time. The goal of compositing is to combine visual content from different sources, such as live-action footage, still images and animations, in a way that they match each other regarding colour, lighting, scale, perspective, camera movement and timing. Most of these aspects can be matched using geometric calibration of the camera and mixed- reality rendering techniques. To ensure that both sources of visual content are composited seamlessly, our approach combines the accuracy of off-line camera tracking with real-time mixed-reality rendering performed in the Unity game engine.