Geometry, Reflectance and Lighting from Neural Rendering

Abstract

This thesis tackles the challenges of 3D geometry, reflectance, and lighting estimation through neural rendering to enhance the modeling and understanding of real-world scenes using computer vision. Existing techniques, while show impressive results, have various limitations, such as the need for specialized hardware, laborious manual annotation, simplified assumptions, excessive input data, or inability to model complex interactions between objects and their environment.

To address these limitations, this thesis presents six novel contributions that leverage deep learning, optimization, and neural rendering techniques, and solve such challenging real-world problems in computer vision and graphics. Our methods can handle diverse factors involved in image formation processes, including unknown lighting, non-Lambertian materials, and geometric interactions.

As the first contribution, the thesis proposes a deep learning framework to minimize the number of input images required for photometric stereo under diverse illumination conditions. The second contribution is a self-supervised coordinate-based deep multilayer perceptron (MLP) network for recovering complex 3D shapes and non-Lambertian reflectance from real-world images. Our third contribution is to jointly optimise object shape, light directions, and light intensities under general surfaces and lighting assumptions to address the un-calibrated photometric stereo problem and the well-known generalized bas-relief (GBR) ambiguity. The fourth contribution presents a 3D compositional morphable model of eyeglasses that accurately incorporates geometric and photometric interaction effects with faces, improving the authenticity of virtual human representations. The fifth study estimates high-definition spatially-varying lighting, reflectance, and geometry of a scene from 360 stereo images, enabling more augmented reality applications. Lastly, the sixth contribution proposes a simple MLP network-based approach to learn the plenoptic function for novel view synthesis from a small number of sparsely configured camera views.

In summary, these contributions extend the boundaries of neural rendering and optimization, opening up new possibilities for advancements in geometry, reflectance, and lighting. By addressing the limitations of previous approaches, this thesis provides a foundation for more robust and accurate real-world scene modeling and understanding. Show more