Video for Virtual Reality

Christian Richardt, James Tompkin, Jordan Halsey, Aaron Hertzmann, Jonathan Starck, Oliver Wang

Research output: Contribution to conferenceOther

  • 1 Citations

Abstract

Video can capture the dynamic appearance of the real world in a way no other technology does; virtual reality technology, on the other hand, enables the display of dynamic visual content with unparalleled realism and immersion. The fusion of these two technologies—video for virtual reality (VR)—promises to enable many exciting photo-realistic experiences.

Over half a day, this course will provide an overview of three aspects of this exciting medium: the technical foundations, current systems in practice, and the potential for future systems of VR video.

In the first section, we will explore the geometric and optical problems underpinning VR video. Then, we will introduce both 360 degree video and stereoscopic video, including how 360 video is captured, analyzed, and stitched, including the mathematics behind how stereo 360 video can be captured.

This background material provides the prerequisites for understanding current systems in use. In the middle hour of the course, we explain how state-of-the-art stereo 360 video is produced from camera systems and computational processing. Then, we will consider the art of storytelling in VR, and how new tools for editing VR video can aid in the craft of this art production. Finally, this section provides an industry perspective covering current production and post-production choices and practice, including CG integration.

The final part of our course focuses on the next generation of video for VR, where we move to 6 degrees-of-freedom (6DoF) experiences. We introduce the basics and challenges behind light field cameras, processing and displays, and see how they can enable 6DoF experiences. This will be followed by another industry perspective on how light field camera arrays have been used to create cutting-edge experiences integrating volumetric live-action elements. To conclude the course, we will see how far we still must go toward the ideal system, in hopes of inspiring the attendees to push the boundary farther to reach it.

We hope this course is useful to a broad audience—at SIGGRAPH and beyond—as we cover the academic, artistic, and production sides of VR video.

Conference

ConferenceSIGGRAPH 2017
Abbreviated titleSIGGRAPH
CountryUSA United States
CityLos Angeles
Period30/07/173/08/17
Internet address

Fingerprint

Virtual reality
Cameras
Display devices
Processing
Industry
Fusion reactions

Cite this

Richardt, C., Tompkin, J., Halsey, J., Hertzmann, A., Starck, J., & Wang, O. (2017). Video for Virtual Reality. SIGGRAPH 2017, Los Angeles, USA United States. https://doi.org/10.1145/3084873.3084894

Video for Virtual Reality. / Richardt, Christian; Tompkin, James; Halsey, Jordan; Hertzmann, Aaron; Starck, Jonathan; Wang, Oliver.

2017. SIGGRAPH 2017, Los Angeles, USA United States.

Research output: Contribution to conferenceOther

Richardt, C, Tompkin, J, Halsey, J, Hertzmann, A, Starck, J & Wang, O 2017, 'Video for Virtual Reality' SIGGRAPH 2017, Los Angeles, USA United States, 30/07/17 - 3/08/17, . https://doi.org/10.1145/3084873.3084894
Richardt C, Tompkin J, Halsey J, Hertzmann A, Starck J, Wang O. Video for Virtual Reality. 2017. SIGGRAPH 2017, Los Angeles, USA United States. https://doi.org/10.1145/3084873.3084894
Richardt, Christian ; Tompkin, James ; Halsey, Jordan ; Hertzmann, Aaron ; Starck, Jonathan ; Wang, Oliver. / Video for Virtual Reality. SIGGRAPH 2017, Los Angeles, USA United States.
@conference{d941295a1d3e40a1bdfadb73d110e88b,
title = "Video for Virtual Reality",
abstract = "Video can capture the dynamic appearance of the real world in a way no other technology does; virtual reality technology, on the other hand, enables the display of dynamic visual content with unparalleled realism and immersion. The fusion of these two technologies—video for virtual reality (VR)—promises to enable many exciting photo-realistic experiences.Over half a day, this course will provide an overview of three aspects of this exciting medium: the technical foundations, current systems in practice, and the potential for future systems of VR video.In the first section, we will explore the geometric and optical problems underpinning VR video. Then, we will introduce both 360 degree video and stereoscopic video, including how 360 video is captured, analyzed, and stitched, including the mathematics behind how stereo 360 video can be captured.This background material provides the prerequisites for understanding current systems in use. In the middle hour of the course, we explain how state-of-the-art stereo 360 video is produced from camera systems and computational processing. Then, we will consider the art of storytelling in VR, and how new tools for editing VR video can aid in the craft of this art production. Finally, this section provides an industry perspective covering current production and post-production choices and practice, including CG integration.The final part of our course focuses on the next generation of video for VR, where we move to 6 degrees-of-freedom (6DoF) experiences. We introduce the basics and challenges behind light field cameras, processing and displays, and see how they can enable 6DoF experiences. This will be followed by another industry perspective on how light field camera arrays have been used to create cutting-edge experiences integrating volumetric live-action elements. To conclude the course, we will see how far we still must go toward the ideal system, in hopes of inspiring the attendees to push the boundary farther to reach it.We hope this course is useful to a broad audience—at SIGGRAPH and beyond—as we cover the academic, artistic, and production sides of VR video.",
author = "Christian Richardt and James Tompkin and Jordan Halsey and Aaron Hertzmann and Jonathan Starck and Oliver Wang",
year = "2017",
month = "7",
day = "30",
doi = "10.1145/3084873.3084894",
language = "English",
note = "SIGGRAPH 2017, SIGGRAPH ; Conference date: 30-07-2017 Through 03-08-2017",
url = "http://s2017.siggraph.org/",

}

TY - CONF

T1 - Video for Virtual Reality

AU - Richardt, Christian

AU - Tompkin, James

AU - Halsey, Jordan

AU - Hertzmann, Aaron

AU - Starck, Jonathan

AU - Wang, Oliver

PY - 2017/7/30

Y1 - 2017/7/30

N2 - Video can capture the dynamic appearance of the real world in a way no other technology does; virtual reality technology, on the other hand, enables the display of dynamic visual content with unparalleled realism and immersion. The fusion of these two technologies—video for virtual reality (VR)—promises to enable many exciting photo-realistic experiences.Over half a day, this course will provide an overview of three aspects of this exciting medium: the technical foundations, current systems in practice, and the potential for future systems of VR video.In the first section, we will explore the geometric and optical problems underpinning VR video. Then, we will introduce both 360 degree video and stereoscopic video, including how 360 video is captured, analyzed, and stitched, including the mathematics behind how stereo 360 video can be captured.This background material provides the prerequisites for understanding current systems in use. In the middle hour of the course, we explain how state-of-the-art stereo 360 video is produced from camera systems and computational processing. Then, we will consider the art of storytelling in VR, and how new tools for editing VR video can aid in the craft of this art production. Finally, this section provides an industry perspective covering current production and post-production choices and practice, including CG integration.The final part of our course focuses on the next generation of video for VR, where we move to 6 degrees-of-freedom (6DoF) experiences. We introduce the basics and challenges behind light field cameras, processing and displays, and see how they can enable 6DoF experiences. This will be followed by another industry perspective on how light field camera arrays have been used to create cutting-edge experiences integrating volumetric live-action elements. To conclude the course, we will see how far we still must go toward the ideal system, in hopes of inspiring the attendees to push the boundary farther to reach it.We hope this course is useful to a broad audience—at SIGGRAPH and beyond—as we cover the academic, artistic, and production sides of VR video.

AB - Video can capture the dynamic appearance of the real world in a way no other technology does; virtual reality technology, on the other hand, enables the display of dynamic visual content with unparalleled realism and immersion. The fusion of these two technologies—video for virtual reality (VR)—promises to enable many exciting photo-realistic experiences.Over half a day, this course will provide an overview of three aspects of this exciting medium: the technical foundations, current systems in practice, and the potential for future systems of VR video.In the first section, we will explore the geometric and optical problems underpinning VR video. Then, we will introduce both 360 degree video and stereoscopic video, including how 360 video is captured, analyzed, and stitched, including the mathematics behind how stereo 360 video can be captured.This background material provides the prerequisites for understanding current systems in use. In the middle hour of the course, we explain how state-of-the-art stereo 360 video is produced from camera systems and computational processing. Then, we will consider the art of storytelling in VR, and how new tools for editing VR video can aid in the craft of this art production. Finally, this section provides an industry perspective covering current production and post-production choices and practice, including CG integration.The final part of our course focuses on the next generation of video for VR, where we move to 6 degrees-of-freedom (6DoF) experiences. We introduce the basics and challenges behind light field cameras, processing and displays, and see how they can enable 6DoF experiences. This will be followed by another industry perspective on how light field camera arrays have been used to create cutting-edge experiences integrating volumetric live-action elements. To conclude the course, we will see how far we still must go toward the ideal system, in hopes of inspiring the attendees to push the boundary farther to reach it.We hope this course is useful to a broad audience—at SIGGRAPH and beyond—as we cover the academic, artistic, and production sides of VR video.

UR - http://richardt.name/video4vr/

U2 - 10.1145/3084873.3084894

DO - 10.1145/3084873.3084894

M3 - Other

ER -