Testing Methods of Testing Replication Success

Abstract

The Replication Crisis refers to a time of doubt on the trustworthiness of the published research findings. Replicability at its simplest form is taken to mean the consistency of a finding. The crisis in confidence in research findings assumes that consistency of a finding confirms an effect and an inconsistent finding disputes an effect. However, the performance of the methods used to test replicability have been largely ignored. Different tests of replicability give different outcomes for the same data, which shows that these tests do not measure the same thing. This thesis explored how sampling variability and study variables (such as sample size, effect size index, base-rates) impact expected rates of replication success and failure. Four approaches to testing replicability were explored using simulation in R: comparing p-values, comparing effect sizes, using heterogeneity tests and comparing Bayes Factors across original and replication studies. The relationship between sampling variability and various study characteristics on the expected outcomes of these tests was complex. Unlike the assumption that consistency in findings confirms an effect and inconsistency in findings disputes an effect, the simulation studies in this thesis demonstrated that under certain combinations of study characteristics, replication success was unlikely to occur and replication failure was likely to occur even though the effect was truly consistent. When a replication failure occurs, it is difficult to interpret whether this is due to true and meaningful inconsistency in effects, or due to other factors unrelated to the consistency of the effect. Replication cannot conclusively verify or disconfirm prior findings due to this difficulty in interpreting replication outcomes. It is suggested that the importance of replication in psychology is in its role in improving research practice, not in its ability to conclusively verify prior results.