The evaluation of information technology projects : a study of effective practices

Abstract

The topic of IT project evaluation is important due to the high cost and strategic importance of many IT projects and long-standing difficulties with their evaluation. While it is widely recognised that the evaluation of IT projects is problematic, there is limited research into how organisations can improve their evaluation practices. The literature supports the divergent views that current IT project evaluation practices are inadequate, more formal and rigorous methods are required, a wide range of techniques is already available, yet very few of the currently available techniques are used in practice. However, there is very little empirical research into what constitutes an appropriate level of formality or rigour, or what specific practices are necessary for evaluation to be effective. This study set out to identify the most effective IT project evaluation practices used by organisations in Australia, and to understand why they work. This exploratory study follows a qualitative theory-building paradigm, where the emerging theory helps explain what is happening in practice. Qualitative analysis of interviews with 72 senior managers in 36 companies in three industries was used to determine effective evaluation practices. Six key dimensions of effective IT project evaluation practice were found to be related to effective IT project evaluation outcomes leading to more efficient use of resources and improved IT project success. The six dimensions were as follows. First, evaluation was effective when there was top-leadership commitment and business engagement. Second, a clear focus was achieved during ex-ante evaluation by aligning projects to strategy and having an agreed definition of project success. Third, control at both a project and portfolio level was enabled by stage gates, portfolio management and dedicated resources. Fourth, effective evaluation processes were scaled to balance governance and responsiveness. Fifth, evaluation and measurement were continuous and integrated. Finally, the use of evaluation results and accountability reinforced the effectiveness of evaluation practices. While these concepts may be discussed in isolation in the extant IT project management literature, few studies present them in an integrated manner and relate them to effective IT project evaluation outcomes and IT project success. The key finding of this study is that more formal evaluation is not necessarily better. In the 36 case study companies, some level of formality helped improve evaluation and, ultimately, IT project outcomes. However, evaluation processes that were too formal were ineffective, resulting in dysfunctional behaviour. Whilst regular evaluation across the project lifecycle the goal of most companies, the key issue was one of implementation of that intention. Many companies had well-documented processes and methods, but they were not applied consistently. Thus, formal processes and methods alone were not enough. It was only when all of the effective practices were combined that positive behaviours were reinforced, actions were aligned, and evaluation processes were most effective. This research contributes to theory development by presenting a substantive theory of effective ITproject evaluation grounded on rich empirical data. The theoretical model developed addresses important gaps in the literature, in particular by identifying which practices are most effective, integrating a range of concepts and relating effective practices to IT project success. This contribution is important due to the lack of recognition to date of effective IT project evaluation practices. The practices identified in this study also provide the foundation for further research into IT project evaluation practices, and the relationships between these practices and project success. These conclusions provide important insights for improving IT evaluation practices, and ultimately, IT project outcomes, both in Australia and around the world.