MatryODShka: Real-time 6DoF Video View Synthesis using Multi-Sphere Images

Benjamin Attal; Selena Ling; Aaron  Gokaslan; Christian Richardt; James Tompkin

doi:10.1007/978-3-030-58452-8_26

MatryODShka: Real-time 6DoF Video View Synthesis using Multi-Sphere Images

Benjamin Attal, Selena Ling, Aaron Gokaslan, Christian Richardt, James Tompkin

Research output: Chapter or section in a book/report/conference proceeding › Chapter in a published conference proceeding

48 Citations (SciVal)

155 Downloads (Pure)

Abstract

We introduce a method to convert stereo 360° (omnidirectional stereo) imagery into a layered, multi-sphere image representation for six degree-of-freedom (6DoF) rendering. Stereo 360° imagery can be captured from multi-camera systems for virtual reality (VR), but lacks motion parallax and correct-in-all-directions disparity cues. Together, these can quickly lead to VR sickness when viewing content. One solution is to try and generate a format suitable for 6DoF rendering, such as by estimating depth. However, this raises questions as to how to handle disoccluded regions in dynamic scenes. Our approach is to simultaneously learn depth and disocclusions via a multi-sphere image representation, which can be rendered with correct 6DoF disparity and motion parallax in VR. This significantly improves comfort for the viewer, and can be inferred and rendered in real time on modern GPU hardware. Together, these move towards making VR video a more comfortable immersive medium.

Original language	English
Title of host publication	Proceedings of the 16th European Conference on Computer Vision (ECCV)
Publisher	Springer
Pages	441-459
Number of pages	19
Volume	2020
ISBN (Electronic)	978-3-030-58452-8
ISBN (Print)	978-3-030-58451-1
DOIs	https://doi.org/10.1007/978-3-030-58452-8_26
Publication status	Published - 3 Nov 2020
Event	European Conference on Computer Vision 2020 - online Duration: 24 Aug 2020 → 28 Aug 2020 https://eccv2020.eu/

Conference

Conference	European Conference on Computer Vision 2020
Abbreviated title	ECCV
Period	24/08/20 → 28/08/20
Internet address	https://eccv2020.eu/

Access to Document

10.1007/978-3-030-58452-8_26

MatryODShka-AttalEtAl-ECCV2020
This is a post-peer-review, pre-copyedit version of an article published in European Conference on Computer Vision - ECCV 2020. The final authenticated version is available online at: https://doi.org/10.1007/978-3-030-58452-8_26
Accepted author manuscript, 8.55 MB

Fellowship - Towards Immersive 360° VR Video with Motion Parallax
Richardt, C. (PI)
Engineering and Physical Sciences Research Council
25/06/18 → 24/12/21
Project: Research council
Centre for the Analysis of Motion, Entertainment Research and Applications (CAMERA)
Cosker, D. (PI), Campbell, N. (CoI), Fincham Haines, T. (CoI), Hall, P. (CoI), Kim, K. I. (CoI), Lutteroth, C. (CoI), O'Neill, E. (CoI), Richardt, C. (CoI) & Yang, Y. (CoI)
Engineering and Physical Sciences Research Council
1/09/15 → 28/02/21
Project: Research council

Cite this

@inproceedings{75299fe1de584c8696677fdb4d523123,

title = "MatryODShka: Real-time 6DoF Video View Synthesis using Multi-Sphere Images",

abstract = "We introduce a method to convert stereo 360° (omnidirectional stereo) imagery into a layered, multi-sphere image representation for six degree-of-freedom (6DoF) rendering. Stereo 360° imagery can be captured from multi-camera systems for virtual reality (VR), but lacks motion parallax and correct-in-all-directions disparity cues. Together, these can quickly lead to VR sickness when viewing content. One solution is to try and generate a format suitable for 6DoF rendering, such as by estimating depth. However, this raises questions as to how to handle disoccluded regions in dynamic scenes. Our approach is to simultaneously learn depth and disocclusions via a multi-sphere image representation, which can be rendered with correct 6DoF disparity and motion parallax in VR. This significantly improves comfort for the viewer, and can be inferred and rendered in real time on modern GPU hardware. Together, these move towards making VR video a more comfortable immersive medium.",

author = "Benjamin Attal and Selena Ling and Aaron Gokaslan and Christian Richardt and James Tompkin",

year = "2020",

month = nov,

day = "3",

doi = "10.1007/978-3-030-58452-8_26",

language = "English",

isbn = "978-3-030-58451-1",

volume = "2020",

pages = "441--459",

booktitle = "Proceedings of the 16th European Conference on Computer Vision (ECCV)",

publisher = "Springer",

note = "European Conference on Computer Vision 2020, ECCV ; Conference date: 24-08-2020 Through 28-08-2020",

url = "https://eccv2020.eu/",

}

TY - GEN

T1 - MatryODShka: Real-time 6DoF Video View Synthesis using Multi-Sphere Images

AU - Attal, Benjamin

AU - Ling, Selena

AU - Gokaslan, Aaron

AU - Richardt, Christian

AU - Tompkin, James

PY - 2020/11/3

Y1 - 2020/11/3

N2 - We introduce a method to convert stereo 360° (omnidirectional stereo) imagery into a layered, multi-sphere image representation for six degree-of-freedom (6DoF) rendering. Stereo 360° imagery can be captured from multi-camera systems for virtual reality (VR), but lacks motion parallax and correct-in-all-directions disparity cues. Together, these can quickly lead to VR sickness when viewing content. One solution is to try and generate a format suitable for 6DoF rendering, such as by estimating depth. However, this raises questions as to how to handle disoccluded regions in dynamic scenes. Our approach is to simultaneously learn depth and disocclusions via a multi-sphere image representation, which can be rendered with correct 6DoF disparity and motion parallax in VR. This significantly improves comfort for the viewer, and can be inferred and rendered in real time on modern GPU hardware. Together, these move towards making VR video a more comfortable immersive medium.

AB - We introduce a method to convert stereo 360° (omnidirectional stereo) imagery into a layered, multi-sphere image representation for six degree-of-freedom (6DoF) rendering. Stereo 360° imagery can be captured from multi-camera systems for virtual reality (VR), but lacks motion parallax and correct-in-all-directions disparity cues. Together, these can quickly lead to VR sickness when viewing content. One solution is to try and generate a format suitable for 6DoF rendering, such as by estimating depth. However, this raises questions as to how to handle disoccluded regions in dynamic scenes. Our approach is to simultaneously learn depth and disocclusions via a multi-sphere image representation, which can be rendered with correct 6DoF disparity and motion parallax in VR. This significantly improves comfort for the viewer, and can be inferred and rendered in real time on modern GPU hardware. Together, these move towards making VR video a more comfortable immersive medium.

U2 - 10.1007/978-3-030-58452-8_26

DO - 10.1007/978-3-030-58452-8_26

M3 - Chapter in a published conference proceeding

SN - 978-3-030-58451-1

VL - 2020

SP - 441

EP - 459

BT - Proceedings of the 16th European Conference on Computer Vision (ECCV)

PB - Springer

T2 - European Conference on Computer Vision 2020

Y2 - 24 August 2020 through 28 August 2020

ER -

MatryODShka: Real-time 6DoF Video View Synthesis using Multi-Sphere Images

Abstract

Conference

Access to Document

Fingerprint

Projects

Fellowship - Towards Immersive 360° VR Video with Motion Parallax

Centre for the Analysis of Motion, Entertainment Research and Applications (CAMERA)

Cite this