How to tell Real From Fake? Understanding how to classify human-authored and machine-generated text

Abstract

Natural Language Generation (NLG) using Generative Adversarial Networks (GANs) has been an active field of research as it alleviates restrictions in conventional Language Modelling based text generators e.g. Long-Short Term Memory (LSTM) networks. The adequacy of a GAN-based text generator depends on its capacity to classify human-written (real) and machine-generated (synthetic) text. However, traditional evaluation metrics used by these generators cannot effectively capture classification features in NLG tasks, such as creative writing. We prove this by using an LSTM network to almost perfectly classify sentences generated by a LeakGAN, a state-of-the-art GAN for long text generation. This thesis attempts a rare approach to understand real and synthetic sentences using meaningful and interpretable features of long sentences (with at least 20 words). We analyse novelty and diversity features of real and synthetic sentences, generate by a LeakGAN, using three meaningful text dissimilarity functions: Jaccard Distance (JD), Normalised Levenshtein Distance (NLD) and Word Mover’s Distance (WMD). In particular, these functions focus on (1) the number of common words, (2) the order of these words, and (3) the semantic similarity in both sentence types, making them interpretable. We provide a comprehensive investigation to identify the effectiveness of novelty and diversity, in classifying real and synthetic sentences, by training two different classification algorithms of varying complexities. Our evaluations show that sentence diversities, using JD and NLD, are the most effective features for classification of human-authored and machine-generated sentences.