Dependent normalized random measures

Abstract

Bayesian nonparametrics, since its introduction, has gained increasing attention in machine learning due to its flexibility in modeling. Essentially, Bayesian nonparametrics defines distributions over infinite dimensional objects such as discrete distributions and smooth functions. This overcomes the fundamental problem of model selection which is hard in traditional machine learning, thus is appealing in both application and theory. Among the Bayesian nonparametric family, random probability measures have played important roles in modern machine learning. They have been used as priors for discrete distributions such as topic distributions in topic models. However, a general treatment and analysis of the random probability measure has not been fully explored in the machine learning community. This thesis introduces the normalized random measure (NRM), built on theories of Poisson processes and completely random measures from the statistical community. Then a family of dependent normalized random measures, including hierarchical normalized random measures, mixed normalized random measures and thinned normalized random measures, are proposed based on the NRM framework to tackle different kinds of dependency modeling problems, {\it e.g.}, hierarchical topic modeling and dynamic topic modeling. In these dependency models, various distributional properties and posterior inference techniques are analyzed based on the general theory of Poisson process partition calculus. The proposed dependent normalized random measure family generalizes some popular dependent nonparametric Bayesian models such as the hierarchical Dirichlet process, and can be easily adapted to different applications. Finally, more generalized dependent random probability measures and possible future work are discussed. To sum up, the contributions of the thesis include: Transfer the theory of the normalized random measure from the statistical to machine learning community. Normalized random measures, which were proposed recently in the statistical community, generalize the Dirichlet process to a large extent, thus are much more flexible in modeling real data. This thesis forms the most extensive research todate in this area. Explore different ideas about constructing dependent normalized random measures. Existing Bayesian nonparametric models only explore limited dependency structures, with probably the most popular and successful being hierarchical construction, {\it e.g.}, the hierarchical Dirichlet process (HDP). The dependency models in the thesis not only extend the HDP to hierarchical normalized random measures for more flexible modeling, but also explore other ideas by controlling specific atoms of the underlying Poisson process. This results in many dependency models with abilities to handle dependencies beyond hierarchical dependency such as the Markovian dependency. In addition, by constructing the dependency models in such ways, various distributional properties and posterior structures can be well analyzed, resulting in much more theoretically clean models. These are lacked of in the hierarchical dependent model. All the models proposed are extensively tested, through both synthetic data and real data, such as in topic modeling of documents. Experimental results have shown superior performance compared to realistic baselines, demonstrating the effectiveness and suitability of the proposed models.