Few-shot Learning for Object Detection

Abstract

The recent developments in neural networks have made deep learning a powerful tool that significantly advances the performance of various computer vision tasks. However, most most deep learning models rely on the large-scale annotated training data, and the performance turns to be low when the size of training data is limited, therefore numbers of large-scale datasets consisting of millions of images are proposed for vision tasks. Though training on large-scale dataset can significantly improve the performance, creating such datasets for novel scenarios is costly. In recent years, researchers have explored few-shot learning in classification task. However, such off-the-shelf few-shot classifiers cannot be directly applied to the FSOD problem which requires simultaneous classification (novel classes) and localization of objects. Few-shot classification is an image-level task where the few-shot learner relies on images of a single object to classify. In contrast, few-shot object detector represents an instance classification problem for which a query image includes multiple objects. The main challenges are multiple disturbed objects not belonging to the given support class and cluttered backgrounds in the query image. These necessitate robustness to large geometric and photometric variances across objects and distinguishing them from disturbances.

We focus on enhancing discriminative feature representations to form accurate relationships across support-query pairs and improving robustness of class-wise prototypes extracted from a few support samples. This thesis makes contributions to FSOD from four different papers. First, we propose using second-order statistics for support-query matching and study a proper way to use co-occurrence statistics. Co-occurrence statistics capture how frequently different objects occur together and help understand the relationship between various visual features in datasets. Initial experiments revealed limitations due to nuisance variability in co-occurrences related to the frequency of certain visual features whose quantity is affected by the scale, pose, and texture areas of objects. To address this, we incorporated power normalization (PN) techniques, significantly improving performance. Following we design a Kernelized Few-shot Object Detector by leveraging kernelized matrices computed over multiple proposal regions, which yield expressive shift-invariant non-linear representations. This shift-invariant property enables the detector to be invariant to the object's physical location, orientation, viewpoint. The model complexity is learned on the fly by designing a learnable RBF kernel radius that controls a complex decision boundary. Forming kernel matrix for hundreds of proposals is costly but that cost is reduced by our proposed Integral Region-of-Interest Aggregation and Count Sketching units. Count sketching, an unsupervised dimensionality reduction technique, is used for computational efficiency. We have proven that it has the favorable property of implicitly performing feature augmentations. Then, we are the first to explore the high-order descriptors for few-shot object detection, which combines second-, third-, and fourth-order patterns. The computational cost is reduced by the proposed novel Tensor Shrinkage Operator (TSO), which also reverses the diffusion of signals in high-order tensors, resulting in accurate high-order tensor estimators. Last but not least, we delves into prototype learning for few-shot object detection. Our method generates high-quality prototypes by prioritizing salient representations and reducing trivial variations within each class. By decoupling prototypes into task-specific ones, the method provides tailored guidance for the Region Proposal Network (RPN) and Detection Head (DH), reducing their conflicts. Show more