Not All Negatives Are Equal: Learning to Track With Multiple Background Clusters

Abstract

Conventional tracking-by-detection approaches for visual object tracking often assume that the task at hand is a binary foreground-versus-background classification problem where the background is a single, generic, and all-inclusive class. In contrast, here we argue that the background appearance for the most part possesses a more complicated structure that will benefit from further partitioning into multiple contextual clusters. Our observation is that, although the background class is contemplated to contain a vast intra-class variation, during the tracking process only a small portion of this variation is present at the current frame around the foreground object. This motivates us to build multiple fine-grained foreground-versuscontextual- cluster models in order to achieve more discriminative classifications, and consequently more robust and accurate foreground object tracking. We learn in an online fashion to optimally fuse the results from multiple classifiers in a principled manner. Structured output support vector machine (SSVM) is employed for each classifier and for fusion. We show that this is not achievable by simply increasing the complexity of a single classifier, i.e. increasing the number of support vectors. Our extensive evaluations on large benchmark datasets demonstrate that our tracker consistently outperforms the current state-ofthe- art while having comparable computational requirements.