Direct, Dense, and Deformable: Template-Based Non-Rigid 3D Reconstruction from RGB Video

Rui Yu, Chris Russell, Neill Campbell, Lourdes Agapito

Research output: Contribution to conferencePaper

  • 19 Citations

Abstract

In this paper we tackle the problem of capturing the dense, detailed 3D geometry of generic, complex non-rigid meshes using a single RGB-only commodity video camera and a direct approach. While robust and even real-time solutions exist to this problem if the observed scene is static, for non-rigid dense shape capture current systems are typically restricted to the use of complex multi-camera rigs, take advantage of the additional depth channel available in RGB-D cameras, or deal with specific shapes such as faces or planar surfaces. In contrast, our method makes use of a single RGB video as input; it can capture the deformations of generic shapes; and the depth estimation is dense, per-pixel and direct. We first compute a dense 3D template of the shape of the object, using a short rigid sequence, and subsequently perform online reconstruction of the non-rigid mesh as it evolves over time. Our energy optimization approach minimizes a robust photometric cost that simultaneously estimates the temporal correspondences and 3D deformations with respect to the template mesh. In our experimental evaluation we show a range of qualitative results on novel datasets; we compare against an existing method that requires multi-frame optical flow; and perform a quantitative evaluation against other template-based approaches on a ground truth dataset.

Conference

ConferenceIEEE International Conference on Computer Vision (ICCV 2015)
CountryUK United Kingdom
Period13/12/15 → …

Fingerprint

Cameras
Optical flows
Video cameras
Pixels
Geometry
Costs

Cite this

Yu, R., Russell, C., Campbell, N., & Agapito, L. (2015). Direct, Dense, and Deformable: Template-Based Non-Rigid 3D Reconstruction from RGB Video. Paper presented at IEEE International Conference on Computer Vision (ICCV 2015), UK United Kingdom.

Direct, Dense, and Deformable : Template-Based Non-Rigid 3D Reconstruction from RGB Video. / Yu, Rui; Russell, Chris; Campbell, Neill; Agapito, Lourdes.

2015. Paper presented at IEEE International Conference on Computer Vision (ICCV 2015), UK United Kingdom.

Research output: Contribution to conferencePaper

Yu, R, Russell, C, Campbell, N & Agapito, L 2015, 'Direct, Dense, and Deformable: Template-Based Non-Rigid 3D Reconstruction from RGB Video' Paper presented at IEEE International Conference on Computer Vision (ICCV 2015), UK United Kingdom, 13/12/15, .
Yu R, Russell C, Campbell N, Agapito L. Direct, Dense, and Deformable: Template-Based Non-Rigid 3D Reconstruction from RGB Video. 2015. Paper presented at IEEE International Conference on Computer Vision (ICCV 2015), UK United Kingdom.
Yu, Rui ; Russell, Chris ; Campbell, Neill ; Agapito, Lourdes. / Direct, Dense, and Deformable : Template-Based Non-Rigid 3D Reconstruction from RGB Video. Paper presented at IEEE International Conference on Computer Vision (ICCV 2015), UK United Kingdom.
@conference{39d0ff5d32db41e9a4030e4b2ab628e1,
title = "Direct, Dense, and Deformable: Template-Based Non-Rigid 3D Reconstruction from RGB Video",
abstract = "In this paper we tackle the problem of capturing the dense, detailed 3D geometry of generic, complex non-rigid meshes using a single RGB-only commodity video camera and a direct approach. While robust and even real-time solutions exist to this problem if the observed scene is static, for non-rigid dense shape capture current systems are typically restricted to the use of complex multi-camera rigs, take advantage of the additional depth channel available in RGB-D cameras, or deal with specific shapes such as faces or planar surfaces. In contrast, our method makes use of a single RGB video as input; it can capture the deformations of generic shapes; and the depth estimation is dense, per-pixel and direct. We first compute a dense 3D template of the shape of the object, using a short rigid sequence, and subsequently perform online reconstruction of the non-rigid mesh as it evolves over time. Our energy optimization approach minimizes a robust photometric cost that simultaneously estimates the temporal correspondences and 3D deformations with respect to the template mesh. In our experimental evaluation we show a range of qualitative results on novel datasets; we compare against an existing method that requires multi-frame optical flow; and perform a quantitative evaluation against other template-based approaches on a ground truth dataset.",
author = "Rui Yu and Chris Russell and Neill Campbell and Lourdes Agapito",
note = "Also published in Proceedings of the IEEE International Conference on Computer Vision, 11-18 December 2015. Article no. 7410468, pp. 918-926; IEEE International Conference on Computer Vision (ICCV 2015) ; Conference date: 13-12-2015",
year = "2015",
month = "12",
language = "English",

}

TY - CONF

T1 - Direct, Dense, and Deformable

T2 - Template-Based Non-Rigid 3D Reconstruction from RGB Video

AU - Yu, Rui

AU - Russell, Chris

AU - Campbell, Neill

AU - Agapito, Lourdes

N1 - Also published in Proceedings of the IEEE International Conference on Computer Vision, 11-18 December 2015. Article no. 7410468, pp. 918-926

PY - 2015/12

Y1 - 2015/12

N2 - In this paper we tackle the problem of capturing the dense, detailed 3D geometry of generic, complex non-rigid meshes using a single RGB-only commodity video camera and a direct approach. While robust and even real-time solutions exist to this problem if the observed scene is static, for non-rigid dense shape capture current systems are typically restricted to the use of complex multi-camera rigs, take advantage of the additional depth channel available in RGB-D cameras, or deal with specific shapes such as faces or planar surfaces. In contrast, our method makes use of a single RGB video as input; it can capture the deformations of generic shapes; and the depth estimation is dense, per-pixel and direct. We first compute a dense 3D template of the shape of the object, using a short rigid sequence, and subsequently perform online reconstruction of the non-rigid mesh as it evolves over time. Our energy optimization approach minimizes a robust photometric cost that simultaneously estimates the temporal correspondences and 3D deformations with respect to the template mesh. In our experimental evaluation we show a range of qualitative results on novel datasets; we compare against an existing method that requires multi-frame optical flow; and perform a quantitative evaluation against other template-based approaches on a ground truth dataset.

AB - In this paper we tackle the problem of capturing the dense, detailed 3D geometry of generic, complex non-rigid meshes using a single RGB-only commodity video camera and a direct approach. While robust and even real-time solutions exist to this problem if the observed scene is static, for non-rigid dense shape capture current systems are typically restricted to the use of complex multi-camera rigs, take advantage of the additional depth channel available in RGB-D cameras, or deal with specific shapes such as faces or planar surfaces. In contrast, our method makes use of a single RGB video as input; it can capture the deformations of generic shapes; and the depth estimation is dense, per-pixel and direct. We first compute a dense 3D template of the shape of the object, using a short rigid sequence, and subsequently perform online reconstruction of the non-rigid mesh as it evolves over time. Our energy optimization approach minimizes a robust photometric cost that simultaneously estimates the temporal correspondences and 3D deformations with respect to the template mesh. In our experimental evaluation we show a range of qualitative results on novel datasets; we compare against an existing method that requires multi-frame optical flow; and perform a quantitative evaluation against other template-based approaches on a ground truth dataset.

M3 - Paper

ER -