Selecting the best meta-analytic estimator for evidence-based practice: a simulation study

Abstract

Studies included in meta-analysis can produce results that depart from the true population parameter of interest due to systematic and/or random errors. Synthesis of these results in meta-analysis aims to generate an estimate closer to the true population parameter by minimizing these errors across studies. The inverse variance heterogeneity (IVhet), quality effects and random effects models of meta-analysis all attempt to do this, but there remains controversy around the estimator that best achieves this goal of reducing error. In an attempt to answer this question, a simulation study was conducted to compare estimator performance. Five thousand iterations at 10 different levels of heterogeneity were run, with each iteration generating one meta-analysis. The results demonstrate that the IVhet and quality effects estimators, though biased, have the lowest mean squared error. These estimators also achieved a coverage probability at or above the nominal level (95%), whereas the coverage probability under the random effects estimator significantly declined (<80%) as heterogeneity increased despite a similar confidence interval width. Based on our findings, we would recommend the use of the IVhet and quality effects models and a discontinuation of traditional random effects models currently in use for meta-analysis