Unsupervised extraction of local image descriptors via relative distance ranking loss

Abstract

State-of-the-art supervised local descriptor learning methods heavily rely on accurately labelled patches for training. However, since the process of labelling patches is laborious and inefficient, supervised training is limited by the availability and scale of training datasets. In comparison, unsupervised learning does not require burdensome data labelling; thus it is not restricted to a specific domain. Furthermore, extracting patches from training images in-volves minimal effort. Nevertheless, most of the existing unsupervised learning based methods are inherently inferior to the handcrafted local descriptors, such as the Scale-Invariant Feature Transform (SIFT). In this paper, we aim to leverage unlabelled data to learn descriptors for image patches by a deep convolutional neural network. We introduce a Relative Distance Ranking Loss (RDRL) that measures the deviation of a generated ranking order of patch similarities against a reference one. Specifically, our approach yields a patch similarity ranking based on the learned embedding of a neural network, and the ranking mechanism minimizes the proposed RDRL by mimicking a reference similarity ranking based on a competent handcrafted feature (i.e., SIFT). To our advantage, after the training process, our network is not only able to measure the patch similarity but also able to outperform SIFT by a large margin on several commonly used benchmark datasets as demonstrated in our extensive experiments.