Egocentric scene reconstruction from an omnidirectional video

Hyeonjoong Jang, Andréas Meuleman, Dahyun Kang, Donggun Kim, Christian Richardt, Min H. Kim

Research output: Contribution to journalArticlepeer-review

5 Citations (SciVal)

Abstract

Omnidirectional videos capture environmental scenes effectively, but they have rarely been used for geometry reconstruction. In this work, we propose an egocentric 3D reconstruction method that can acquire scene geometry with high accuracy from a short egocentric omnidirectional video. To this end, we first estimate per-frame depth using a spherical disparity network. We then fuse per-frame depth estimates into a novel spherical binoctree data structure that is specifically designed to tolerate spherical depth estimation errors. By subdividing the spherical space into binary tree and octree nodes that represent spherical frustums adaptively, the spherical binoctree effectively enables egocentric surface geometry reconstruction for environmental scenes while simultaneously assigning high-resolution nodes for closely observed surfaces. This allows to reconstruct an entire scene from a short video captured with a small camera trajectory. Experimental results validate the effectiveness and accuracy of our approach for reconstructing the 3D geometry of environmental scenes from short egocentric omnidirectional video inputs. We further demonstrate various applications using a conventional omnidirectional camera, including novel-view synthesis, object insertion, and relighting of scenes using reconstructed 3D models with texture.

Original languageEnglish
Article number100
Pages (from-to)1-12
JournalACM Transactions on Graphics
Volume41
Issue number4
Early online date22 Jul 2022
DOIs
Publication statusPublished - 31 Jul 2022
Externally publishedYes

Bibliographical note

Funding Information:
We thank the reviewers for their valuable feedback that has helped to improve our paper. Min H. Kim acknowledges the MSIT/IITP of Korea (RS-2022-00155620 and 2017-0-00072) and the Samsung Research Funding Center (SRFC-IT2001-04) for developing partial 3D imaging algorithms, in addition to the support of the NIRCH of Korea (2021A02P02-001), Samsung Electronics, and Microsoft Research Asia. Christian Richardt acknowledges an EPSRC-UKRI Innovation Fellowship (EP/S001050/1) and RCUK grant CAMERA (EP/M023281/1, EP/T022523/1).

Publisher Copyright:
© 2022 Owner/Author.

Keywords

  • 360° video
  • 3D reconstruction
  • Binoctree
  • Spherical disparity
  • TSDF fusion

ASJC Scopus subject areas

  • Computer Graphics and Computer-Aided Design

Fingerprint

Dive into the research topics of 'Egocentric scene reconstruction from an omnidirectional video'. Together they form a unique fingerprint.

Cite this