Rigid body motion segmentation with an RGB-D camera

Abstract

Affordable RGB-D cameras that provide color and depth images of a scene are now commonplace. This thesis investigates the problem of rigid body motion segmentation using such a camera, which is of both practical and theoretical significance. Our focus is on developing novel algorithms that use the rich 3D and color input data, and works irrespective of whether the camera is moving or not. We begin with sparse point track segmentation. Noting the fact that the motion of any two points on a rigid body preserves the Euclidean distance between them over time, a novel binary similarity measure for rigid body motion segmentation is proposed. By using this similarity measure to construct a sparse similarity graph and subsequently extracting connected components from the graph, a fast motion segmentation algorithm that can segment large translational motions is introduced. This is then extended to handle general motion and outliers by proposing a novel motion segmentation algorithm that constructs a dense similarity graph and extracts all possible rigid motion groups as maximal cliques of the graph. Theoretical relationships between maximal cliques of the similarity graph and inliers to Euclidean transformation hypotheses are explored. Given all the rigid motion groups, motion segmentation is cast as a max k-cover problem. We demonstrate the fast performance of the method for sufficiently large motions and moderate number of point tracks. We also investigate how the binary similarity measure or a weighted similarity measure constructed in a similar manner can be used to improve the motion model generation stage of existing motion segmentation methods. A novel guided sampling scheme termed Rigidity-GS which improves the probability of a data sample being an all-inlier sample is proposed. The method is demonstrated to outperform random sampling and produce competitive results with the state-of-the-art. Next we investigate how the Euclidean transformation of an object between two views can be uniquely estimated from a single surface point correspondence using differential geometry concepts and how it can be used in motion segmentation. The method is facilitated by two novel methods to remove the sign ambiguity of the first principal curvature direction. It is shown that under low noise and when multiple motions are involved, the proposed method permits better motion segmentation results than the existing three point Euclidean transformation estimation method. Using any of the above sparse motion segmentation methods, we then investigate how a dense segmentation of an RGB-D image can be obtained. Using alignment errors between point clouds under different motion labels, a dense CRF-based multi-labeling segmentation method is proposed. We demonstrate the successful performance of the method using real RGB-D data. In addition to the above contributions, we also investigate how a 3D reconstruction of a scene can be segmented based on motion. Specifically, we consider a reconstruction given in a truncated signed distance function based volumetric surface representation. By drawing from the above sparse and dense motion segmentation works, we propose two novel methods of segmenting such reconstructions. Unlike the existing method to segment such reconstructions, our methods allow segmenting an object even when it is only partially moved. Show more