BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images

Thu Nguyen Phuoc, Christian Richardt, Long Mai, Yongliang Yang, Niloy J. Mitra

Research output: Chapter or section in a book/report/conference proceedingChapter in a published conference proceeding

54 Downloads (Pure)


We present BlockGAN, an image generative model that learns object-aware 3D scene representations directly from unlabelled 2D images. Current work on scene representation learning either ignores scene background or treats the whole scene as one object. Meanwhile, work that considers scene compositionality treats scene objects only as image patches or 2D layers with alpha maps. Inspired by the computer graphics pipeline, we design BlockGAN to learn to first generate 3D features of background and foreground objects, then combine them into 3D features for the whole scene, and finally render them into realistic images. This allows BlockGAN to reason over occlusion and interaction between objects’ appearance, such as shadow and lighting, and provides control over each object’s 3D pose and identity, while maintaining image realism. BlockGAN is trained end-to-end, using only unlabelled single images, without the need for 3D geometry, pose labels, object masks, or multiple views of the same scene. Our experiments show that using explicit 3D features to represent objects allows BlockGAN to learn disentangled representations both in terms of objects (foreground and background) and their properties (pose and identity).

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 665992
Original languageEnglish
Title of host publication34th Conference on Neural Information Processing Systems
Subtitle of host publicationNeurIPS 2020
Place of PublicationVancouver
PublisherNeural Information Processing Systems Foundation, Inc.
Number of pages12
Publication statusPublished - 12 Dec 2020
EventNeurIPS 2020: Conference on Neural Information Processing Systems -
Duration: 6 Dec 202012 Dec 2020


ConferenceNeurIPS 2020: Conference on Neural Information Processing Systems


Dive into the research topics of 'BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images'. Together they form a unique fingerprint.

Cite this