From Completion to Generation: Learning Representation for 3D Content Enhancement and Creation

Abstract

As the demand for high-quality 3D content continues to grow, the enhancement and creation of 3D models have become crucial for a wide range of applications. Despite significant progress in this field, current methods still face critical challenges, including reliance on difficult-to-obtain labeled data and limitations in both the quality and diversity of generated 3D content. This thesis addresses these challenges by introducing a series of novel methods aimed at advancing the capabilities of 3D content enhancement and generation.

We begin by enhancing existing 3D content collected from the real world. 3D acquisition devices, such as LiDAR and RGB-D cameras, often capture 3D data as point clouds, which are typically sparse and incomplete. To improve the quality of this data for downstream applications, point cloud completion is commonly used as a preprocessing step. In this context, we propose an unsupervised point cloud completion method with a novel generative modeling framework that captures the uncertainty inherent in the completion process. Additionally, we identify the issue of data scarcity in the completion task and introduce a novel self-supervised point cloud completion scheme that relies only on single point cloud observations per object to learn how to enhance the data. Furthermore, we extend 3D point cloud completion into the domain of 3D shape generation by framing it as a conditional generation task. Specifically, we propose a unified method that integrates 3D shape completion, reconstruction, and generation into a single probabilistic framework. Finally, we scale up this unified generation framework to enable the creation of diverse, high-quality 3D assets from a single-view image, seamlessly transforming human imagination into intricate 3D digital content.

In conclusion, this thesis presents contributions that address key challenges in 3D content enhancement and creation. From unsupervised point cloud completion to large-scale 3D shape generation, we introduce novel techniques that push the boundaries of 3D computer vision and lay the foundation for future research in 3D data enhancement and creation.