An investigation into the validity and reliability of the AcciMap approach

Abstract

The aim of this thesis is to investigate the validity and reliability of the AcciMap approach, a systems-based technique for analysing the causes of organisational accidents. This approach has been used to analyse accidents in a number of complex systems and to identify areas in which safety interventions should be directed. However, while the technique is implicitly assumed to be valid and reliable, the questions of whether or not it does, in fact, allow analysts to identify the causes of accidents correctly and whether or not the results obtained are consistent and replicable, have not been addressed. These questions are of critical importance when the findings of AcciMap analyses are used to determine the corrective actions to be taken after an accident, since the safety of the system may be jeopardised if problems are not correctly identified and remedied. In the investigation into the validity and reliability of this technique, a study was performed in which several participants independently analysed an accident, using AcciMap guidelines developed during this research. The aim of the study was to enable the validity of the participants’ results (assessed against results produced by AcciMap experts), the reliability of their results (assessed by comparing participants' findings with those of one another) and the nature and significance of any observed variations in these results, to be examined. The qualitative and quantitative analysis of the results obtained in this study revealed that, although similarities existed between their findings, each participant’s results differed from those of the experts and the other participants. Examination of the nature and significance of these differences indicated that some were insignificant with respect to the meaning portrayed or the potential outcomes of analyses, while others were significant in these terms but could feasibly be eliminated if changes were made to the analysis process. Several observed variations, however, were both significant in these respects and arguably unavoidable, stemming from parts of the analysis requiring subjective analyst judgement and areas in which human error or differences in interpretation were possible. The existence of such variations demonstrates that AcciMap analyses do not always produce entirely valid and reliable results.