Lasso regression: estimation and shrinkage via the limit of Gibbs sampling

Abstract

The application of the lasso is espoused in high dimensional settings where only a small number of the regression coefficients are believed to be non-zero (i.e. the solution is sparse). Moreover, statistical properties of high dimensional lasso estimators are often proved under the assumption that the correlation between the predictors is bounded. In this vein, co-ordinatewise methods, which are the most common means of computing the lasso solution, naturally work well in the presence of low-to-moderate multicollinearity. The computational speed of co-ordinatewise algorithms, although excellent for sparse and low-to-moderate multicollinearity settings, degrades as sparsity decreases and multicollinearity increases. Though lack of sparsity and high multicollinearity can be quite common in contemporary applications, model selection is still a necessity in such settings. Motivated by the limitations of co-ordinatewise algorithms in such ‘non-sparse’ and ‘high multicollinearity’ settings, we propose the novel ‘deterministic Bayesian lasso’ algorithm for computing the lasso solution. This algorithm is developed by considering a limiting version of the Bayesian lasso. In contrast with co-ordinatewise algorithms, the performance of the deterministic Bayesian lasso improves as sparsity decreases and multicollinearity increases. Importantly, in non-sparse and high multicollinearity settings the algorithm proposed can offer substantial increases in computational speed over co-ordinatewise algorithms. A rigorous theoretical analysis demonstrates that the deterministic Bayesian lasso algorithm converges to the lasso solution and it leads to a representation of the lasso estimator which shows how it achieves both inline image- and inline image-types of shrinkage simultaneously. Connections between the deterministic Bayesian lasso and other algorithms are also provided. The benefits of the deterministic Bayesian lasso algorithm are then illustrated on simulated and real data.