OmniMR:: Omnidirectional Mixed Reality with Spatially-Varying Environment Reflections from Moving 360° Video Cameras

Joanna Tarko; James Tompkin; Christian Richardt

doi:10.1109/VR.2019.8798067

OmniMR: Omnidirectional Mixed Reality with Spatially-Varying Environment Reflections from Moving 360° Video Cameras

Joanna Tarko, James Tompkin, Christian Richardt

Research output: Contribution to conference › Poster › peer-review

8 Citations (SciVal)

Abstract

We propose a new approach for creating omnidirectional mixed reality (OmniMR) from moving-camera 360° video. To insert virtual computer-generated elements into a moving-camera 360° video, we reconstruct camera motion and sparse scene content via structure from motion on stitched equirectangular video (the default output format of current 360° cameras). Then, to plausibly reproduce real-world lighting conditions for these inserted elements, we employ inverse tone mapping to recover high dynamic range environment maps which vary spatially along the camera path. We implement our approach into the Unity rendering engine for real-time object rendering with dynamic lighting and user interaction. This expands the use and flexibility of 360° video for mixed reality.

Original language	English
Pages	1177-1178
Number of pages	2
DOIs	https://doi.org/10.1109/VR.2019.8798067
Publication status	Published - 23 Mar 2019
Event	IEEE Conference on Virtual Reality and 3D User Interfaces - Osaka, Japan Duration: 23 Mar 2019 → 27 Mar 2019 Conference number: 26 http://ieeevr.org/2019/

Conference

Conference	IEEE Conference on Virtual Reality and 3D User Interfaces
Abbreviated title	IEEE VR
Country/Territory	Japan
City	Osaka
Period	23/03/19 → 27/03/19
Internet address	http://ieeevr.org/2019/

Keywords

3D reconstruction
Environment maps
Lighting estimation
Mixed reality
Omnidirectional video
Structure from motion

ASJC Scopus subject areas

Human-Computer Interaction
Media Technology

Access to Document

10.1109/VR.2019.8798067

Cite this

@conference{9868e09da4f94b82a93c97ecdf8b2339,

title = "OmniMR:: Omnidirectional Mixed Reality with Spatially-Varying Environment Reflections from Moving 360° Video Cameras",

abstract = "We propose a new approach for creating omnidirectional mixed reality (OmniMR) from moving-camera 360° video. To insert virtual computer-generated elements into a moving-camera 360° video, we reconstruct camera motion and sparse scene content via structure from motion on stitched equirectangular video (the default output format of current 360° cameras). Then, to plausibly reproduce real-world lighting conditions for these inserted elements, we employ inverse tone mapping to recover high dynamic range environment maps which vary spatially along the camera path. We implement our approach into the Unity rendering engine for real-time object rendering with dynamic lighting and user interaction. This expands the use and flexibility of 360° video for mixed reality.",

keywords = "3D reconstruction, Environment maps, Lighting estimation, Mixed reality, Omnidirectional video, Structure from motion",

author = "Joanna Tarko and James Tompkin and Christian Richardt",

year = "2019",

month = mar,

day = "23",

doi = "10.1109/VR.2019.8798067",

language = "English",

pages = "1177--1178",

note = "IEEE Conference on Virtual Reality and 3D User Interfaces, IEEE VR ; Conference date: 23-03-2019 Through 27-03-2019",

url = "http://ieeevr.org/2019/",

}

TY - CONF

T1 - OmniMR:

T2 - IEEE Conference on Virtual Reality and 3D User Interfaces

AU - Tarko, Joanna

AU - Tompkin, James

AU - Richardt, Christian

N1 - Conference code: 26

PY - 2019/3/23

Y1 - 2019/3/23

N2 - We propose a new approach for creating omnidirectional mixed reality (OmniMR) from moving-camera 360° video. To insert virtual computer-generated elements into a moving-camera 360° video, we reconstruct camera motion and sparse scene content via structure from motion on stitched equirectangular video (the default output format of current 360° cameras). Then, to plausibly reproduce real-world lighting conditions for these inserted elements, we employ inverse tone mapping to recover high dynamic range environment maps which vary spatially along the camera path. We implement our approach into the Unity rendering engine for real-time object rendering with dynamic lighting and user interaction. This expands the use and flexibility of 360° video for mixed reality.

AB - We propose a new approach for creating omnidirectional mixed reality (OmniMR) from moving-camera 360° video. To insert virtual computer-generated elements into a moving-camera 360° video, we reconstruct camera motion and sparse scene content via structure from motion on stitched equirectangular video (the default output format of current 360° cameras). Then, to plausibly reproduce real-world lighting conditions for these inserted elements, we employ inverse tone mapping to recover high dynamic range environment maps which vary spatially along the camera path. We implement our approach into the Unity rendering engine for real-time object rendering with dynamic lighting and user interaction. This expands the use and flexibility of 360° video for mixed reality.

KW - 3D reconstruction

KW - Environment maps

KW - Lighting estimation

KW - Mixed reality

KW - Omnidirectional video

KW - Structure from motion

UR - http://www.scopus.com/inward/record.url?scp=85071832784&partnerID=8YFLogxK

U2 - 10.1109/VR.2019.8798067

DO - 10.1109/VR.2019.8798067

M3 - Poster

SP - 1177

EP - 1178

Y2 - 23 March 2019 through 27 March 2019

ER -

OmniMR: Omnidirectional Mixed Reality with Spatially-Varying Environment Reflections from Moving 360° Video Cameras

Abstract

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Fellowship - Towards Immersive 360° VR Video with Motion Parallax

Centre for the Analysis of Motion, Entertainment Research and Applications (CAMERA)

EPSRC Centre for Doctoral Training in Digital Entertainment

Cite this

OmniMR: Omnidirectional Mixed Reality with Spatially-Varying Environment Reflections from Moving 360° Video Cameras

Abstract

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Projects

Fellowship - Towards Immersive 360° VR Video with Motion Parallax

Centre for the Analysis of Motion, Entertainment Research and Applications (CAMERA)

EPSRC Centre for Doctoral Training in Digital Entertainment

Cite this