Estimating reflectance, illumination, and shape from a single view

Abstract

This work investigates a core problem in computer vision: obtaining, from a single view, the geometric and photometric information of a scene - the reflectance properties of objects, their shapes, and the illumination direction. We focus on two important aspects of this problem: (a) adopting a general formulation which encompasses a number of physical phenomena and addresses a wide range of non-Lambertian surfaces, and (b) keeping the number of required input images and prior information as small as possible. There are three main contributions in this work. The first of these is the simultaneous recovery, from a single spectral image, of the surface reflectance properties and shape of objects in a scene. Although the general trend in the literature has been to employ trichromatic imagery and develop algorithms based on a specific reflectance model, we make use of the rich information content of spectral imagery and formulate the problem more generally. The spectral approach allows us to estimate the wavelength-dependent physical properties of objects, and the general formulation allows us to address a wide range of surfaces. In this way, we begin with a general, physical interpretation of the reflection process and cast the recovery of reflectance parameters and shape in terms of a structural optimisation. We produce results on synthetic images, and illustrate how the recovered photometric parameters can be applied to real world imagery for skin recognition. The second contribution addresses the problem, again from a single spectral image, of estimating the direction of illumination in a scene without having any prior knowledge of the reflectance properties or object shapes. Although specular highlights provide strong cues for determining illumination direction, most existing approaches require calibration targets, which limits their applicability. We overcome this limitation and determine the direction of the light source by placing two novel constraints on the specular highlights: coplanarity and Kullback-Leibler divergence. To do this, we start with a general formulation that models the scene radiance as a linear combination of specular and diffuse reflections. This permits the reflection parameters to be recovered through an iterative optimisation, which we render well-posed by adopting a novel reparameterisation. Once the reflectance parameters are in hand, we recover the direction of the single point-light source from the specular reflection and the 3D shape from the diffuse reflection. The third contribution of this thesis is to create a variant of colour photometric stereo which addresses two difficulties encountered in previous approaches: non-Lambertian reflectance and spatially varying albedo. We show that, under complementary coloured illumination, the observed colour of an object varies according to different reflections, which provides clues for estimating shape. To deal with multicoloured surfaces, we propose a colour-correction method which exploits the addition principle of complementary colours, allowing an object's true colour to be estimated. In addition, we present a segmentation method that utilises the colour difference between input images to detect diffuse reflections, specularities, and attached shadows. Finally, we employ an iterative optimisation based on the Torrance-Sparrow reflectance model to address non-Lambertian reflectance.