Shape Clustering and Spatial-temporal Constraint for Non-rigid Structure from Motion

Abstract

Non-rigid Structure-from-Motion (NRSfM) is an active research eld in computer vision. The task of NRSfM is to simultaneously recover camera motion and 3D structure from 2D tracks of a deformable object. This problem is generally categorized into sparse and dense cases in terms of scale, where sparse NRSfM deals with a few feature tracks and dense NRSfM recovers the 3D position of each pixel in an image ow. As NRSfM is essentially an under-constrained problem, recent research has focused on enforcing priors to reliably solve the problem. In this thesis, we propose a shape clustering method for sparse NRSfM and a spatial-temporal constraint for dense NRSfM. For sparse NRSfM, we rst revisit the concept of \reconstructability", which indicates the possibility of reconstructing a 3D shape, given 2D feature tracks and camera motion. We give an extension to it and de ne \reconstructability" from 3D shape complexity and motion complexity. To increase global reconstructability, we then propose an iterative shape clustering method to divide a sequence into several sub-sequences, thus decreasing the shape complexity of each sub-sequence, which is much easier to solve individually. Our method aims at solving the longterm, complex motions, which have been a di cult task for previous methods. Experimental results show that our method outperforms the current state-of-theart methods by a margin, thus pushing the limit of sparse NRSfM. For dense NRSfM, we rst revisit the temporal smoothness utilized in sparse NRSfM and demonstrate that it can be employed for dense case directly. Secondly, we propose a spatial smoothness constraint by enforcing a Laplacian lter to the shape matrix. Finally, to handle real world noise and outliers in measurements, we robustify the data term by using the L1 norm. Our method gives a simple yet elegant convex least-squares optimization, which can be e ectively solved by gradient descent. Experimental results on both synthetic and real images show that the proposed method achieves state-of-the-art performance in dense NRSfM.