BOB: Bayesian optimized bootstrap for approximate posterior sampling in Gaussian mixture models

Abstract

The posterior distribution of a Gaussian mixture model (GMM) provides a natural framework to infer the model parameters or make predictions about a population of interest. That said, sampling from the posterior distribution of GMMs via standard Markov chain Monte Carlo (MCMC) imposes several computational challenges, which have slowed down the adoption of a broader full Bayesian implementation of these models. A growing body of literature has introduced the weighted likelihood bootstrap and the weighted Bayesian bootstrap as alternatives to MCMC sampling. The core idea of these methods is to repeatedly compute maximum a posteriori (MAP) estimates from many randomly weighted posterior densities. These MAP estimates then can be treated as approximate posterior draws. Nonetheless, a central question remains unanswered: How to select the random weights under arbitrary sample sizes. We, therefore, introduce the Bayesian optimized bootstrap (BOB), a computational method to automatically tune these random weights by minimizing, through Bayesian optimization, a black-box and noisy version of the reverse Kullback–Leibler (KL) divergence between the Bayesian posterior and an approximate posterior obtained via random weighting. Our proposed method outperforms competing approaches in recovering the Bayesian posterior, while retaining key asymptotic properties from established methodologies. BOB’s performance is demonstrated through extensive simulations, along with real-world data analyses.