Understanding and Improving Generative Adversarial Networks in Low-Data Settings

Abstract

Generative Adversarial Networks (GANs) have emerged as a powerful tool in machine learning, excelling at generating realistic data through adversarial training between a generator and a discriminator. Despite their success, training GANs remains challenging, particularly in low-data settings, where issues such as discriminator overfitting and unstable convergence arise. These challenges are amplified by the scarcity of available training data, which hinders generalization and leads to poor performance on unseen data.

While extensive research has been conducted to improve GANs in limited data regimes, existing approaches often fail to provide a deeper theoretical understanding of the underlying causes of these challenges, particularly from the perspective of gradient control. In this thesis, we address this gap by focusing on how controlling gradients during training can improve both stability and generalization of GANs in data-efficient scenarios. Specifically, we explore how different techniques for gradient control can mitigate overfitting and improve stability, providing novel insights into the relationship between gradient behavior and GAN performance.

This thesis is organized into five main chapters. The first four chapters tackle different aspects of gradient control in GANs and the final chapter applies gradient insights from the preceding four to effectively fine-tune large foundation models. Chapter 1 identifies the challenges of discriminator overfitting and unstable training in Generative Adversarial Networks (GANs) under limited data conditions, highlighting the need for effective solutions to these issues. Chapter 2 introduces a manifold learning approach that constrains the discriminator's capacity, helping to control gradient magnitudes and improve GAN stability. Chapter 3 presents a novel regularization technique using noise modulation on discriminator feature to prevent discriminator overfitting, and using consistency regularization to reduce gradient explosion and stabilize training. Chapter 4 proposes a Lipschitz-constrained normalization technique that harnesses generalization benefits from Batch Normalization (BN), while also prevents gradient explosion issues of BN during training, demonstrating its effectiveness in enhancing generalization while maintaining computational efficiency. Chapter 5 extends theories from Chapters 3 and 4, introducing a novel consistency regularization that significantly enhances generalization and prevents knowledge forgetting when fine-tuning large foundation models.

In conclusion, this thesis demonstrates that gradient control is pivotal for improving GAN stability and generalization, as well as enhancing transfer learning generalization, particularly in limited-data settings. Moreover, the insights gained from this research have broader implications for other machine learning methods, highlighting the importance of managing gradients to enhance generalization and convergence across various domains. Show more