Label Shift Problem for Image Classification Tasks -- A Comprehensive Analysis

Abstract

The real world deployment of machine learning models usually faces problems of distribution shift.Label shift is a common type of distribution shift that happens in classification tasks, where the source domain (train set) and target domain (test set) have different label distributions p(y) but identical distributions of image given label p(x|y). Under label shift, the classifier trained on the source domain may not perform as well in target domain.

Three problems arise under label shift: 1) detection: detect if label shift occurs, 2) estimation: estimate the target label distribution when no target domain labels are available and 3) correction: adapt a source domain classifier to the target domain under label shift. In practical problems, a label shift estimation model is first deployed to obtain an estimate of the target domain label distribution. A label shift correction model is then used to obtain the desired target domain classifier based on the estimate of the target label distribution.

This thesis investigates the label shift problems in the closed set classification, open set classification and zero-shot classification tasks.

In the closed set classification setting, a novel classifier based label shift estimation model and a more general feature based label shift estimation model are proposed. Based on reasonable assumptions, the two proposed models estimate label shift through EM algorithms, which are proved to converge to a Maximum Likelihood Estimate or Maximum a Posteriori estimate of the target label distribution. The estimation results are then used in a label shift correction model to obtain a target domain classifier without retraining or fine-tuning the original source domain classifier.

In the open set classification setting, a novel classifier based label shift estimation and correction model is proposed. With the help of a reference out-of-distribution dataset at test time, the proposed model estimates the target domain label distribution for both in-distribution classes that appear in the source domain as well as the extra out-of-distribution class. Under reasonable assumptions, the proposed model is proved to converge to a Maximum Likelihood Estimate of the target domain label distribution. The estimation result is then used in a label shift correction model to obtain a target domain classifier based on a pre-trained source domain in-distribution class classifier and a out-of-distribution binary classifier without re-training or fine-tuning.

In the zero-shot learning setting, a label shift correction model is proposed. A novel class-balanced triplet loss is proposed to help cluster features of each class equally when the source domain has an imbalanced label distribution. A Gaussian Process Regression model is then proposed to predict feature prototypes of unseen classes based on feature prototypes of seen classes. The final zero-shot learning classifier is then constructed, which is robust to source domain label shift. Show more