MatryODShka: Real-time 6DoF Video View Synthesis using Multi-Sphere Images

Benjamin Attal, Selena Ling, Aaron Gokaslan, Christian Richardt, James Tompkin

Research output: Chapter in Book/Report/Conference proceedingConference contribution

64 Downloads (Pure)

Abstract

We introduce a method to convert stereo 360° (omnidirectional stereo) imagery into a layered, multi-sphere image representation for six degree-of-freedom (6DoF) rendering. Stereo 360° imagery can be captured from multi-camera systems for virtual reality (VR), but lacks motion parallax and correct-in-all-directions disparity cues. Together, these can quickly lead to VR sickness when viewing content. One solution is to try and generate a format suitable for 6DoF rendering, such as by estimating depth. However, this raises questions as to how to handle disoccluded regions in dynamic scenes. Our approach is to simultaneously learn depth and disocclusions via a multi-sphere image representation, which can be rendered with correct 6DoF disparity and motion parallax in VR. This significantly improves comfort for the viewer, and can be inferred and rendered in real time on modern GPU hardware. Together, these move towards making VR video a more comfortable immersive medium.
Original languageEnglish
Title of host publicationProceedings of the 16th European Conference on Computer Vision (ECCV)
PublisherSpringer
Pages441-459
Number of pages19
Volume2020
ISBN (Electronic)978-3-030-58452-8
ISBN (Print)978-3-030-58451-1
DOIs
Publication statusPublished - 3 Nov 2020
EventEuropean Conference on Computer Vision 2020 - online
Duration: 24 Aug 202028 Aug 2020
https://eccv2020.eu/

Conference

ConferenceEuropean Conference on Computer Vision 2020
Abbreviated titleECCV
Period24/08/2028/08/20
Internet address

Fingerprint

Dive into the research topics of 'MatryODShka: Real-time 6DoF Video View Synthesis using Multi-Sphere Images'. Together they form a unique fingerprint.

Cite this