'Testing for Analysts' Bias in Crime Prevention Experiments: Can We Accept Eisner's One-Tailed Test?

Abstract

Eisner (Journal of Experimental Criminology, this issue, 2009) suggests that developer-led evaluations often make programs look better than independent evaluations do because the former suffer systematic biases in favor of prevention success. Yet, his proposed remedies suffer their own systematic bias, constituting a 'one-tailed' test of bias in only one direction. In this response we suggest that a more objective assessment of 'analysts' effects' requires a 'two-tailed' test of bias, in which reviewers would measure indications of bias both for and against success in evaluations reported by both developers and independent evaluators. After exploring the full complexity of the distinction between developers and evaluators, we report on one case in which independent evaluations were more favorable than those of developers. We then suggest possible indicators of analysts' biases against finding success that may characterize the work of developers who "bend over backwards" to find harm in their programs, and of independent evaluators who may seek to "get a scalp" of a developer or a program.