Bias Assessment in Outcomes Research: The Role of Relative Versus Absolute Approaches

Abstract

Objectives: Bias assessment tools vary in content and detail, and the method used for assessment may produce different assessment results in a study if not carefully considered. Therefore, taking an approach to the assessment of studies that produces a similar result regardless of the tool used for assessment (tool independence) is important. Methods: A preexisting study that used 25 different quality scales was assessed to examine tool dependence of 2 common approaches to bias assessments—absolute value judgments (defined as the qualitative risk of bias judgment based on a threshold across studies) and relative ranks (defined as the relative probability toward bias of a study relative to the best assessed study). Agreement between each of the 25 scales and a composite scale (that includes all unique safeguards across all scales) was computed (using the intraclass correlation coefficient [ICC]; consistency). Tool dependence was considered present when the ICCs were inconsistent across the 25 scales for the same study. Results: We found that using relative ranks for tools with different numbers and types of items produced consistent results, with only small differences in the agreement for the various tools with the composite tool, whereas consistency (measured by the ICC) varied considerably when using absolute judgments. Inconsistency is problematic because it means that the assessment result is linked to the scale and not to the study. Conclusions: Tool independence is an important attribute of a bias assessment tool. On the basis of this study, the use of relative ranks retains tool independence and therefore produces consistent ranks for the same study across tools.