Label-Efficient Learning For Scene Segmentation

Abstract

Data scarcity has emerged as a significant challenge in the large scaling of deep learning systems. This thesis addresses this critical challenge of label scarcity in training object segmentation models, a task often hindered by the need for extensive and costly manual annotation. Our research explores two complementary strategies towards improving label efficiency for training deep models: exploiting implicit information within image data for label-free learning and extending existing labelled data through generative data augmentation.

We develop a fully unsupervised framework for video object segmentation by leveraging a layered scene model that establishes motion-defined ``objectness". We train our network using only unlabelled pairs of image sequences, effectively segmenting the dominant moving object without any ground truth labels. This approach demonstrates the potential of extracting implicit information directly from video data to enable label-free learning.

We then shift focus towards bolstering available training data using synthetically generated instances while ensuring that our framework is robustly scalable. We demonstrate our image-based augmentation approach on the challenging task of small smoke segmentation by extrapolating existing smoke scenes through image out-painting, which simulates a radial camera movement. This process preserves existing labels while generating new semantically coherent training data, leading to substantial improvements in segmentation performance. We then consolidate our findings and expand the scope of our exploration of generative data augmentation by proposing a paradigm of novel object instance creation for dense video tasks. A text-to-video generative model is utilised to synthesise temporally dynamic object instances, which are then automatically segmented, curated and seamlessly integrated into existing video scenes. This automated pipeline significantly enhances segmentation performance across multiple baseline models without requiring any additional manual dense labelling.

Demonstrating the efficacy of both implicit information extraction as well as generative data augmentation in reducing the reliance on manual annotation, we hope to pave the way for more label-efficient training of scalable object segmentation models. Show more