Average evaluation intensity: a quality-oriented indicator for the evaluation of research performance

Abstract

A variety of indicators have been created to measure the research performance of journals, scientists, and institutions. There has been a long-running debate on the use of indicators based on citation counts to measure research quality. The key argument is that using indicators based on raw citation counts to evaluate research quality lacks measurement validity. Traditional reference formats do not present any quality related evaluations of the citing authors toward their references. It can be argued that the strength of peer evaluation to a research output, which is taken to represent its quality, is the elementary unit in the evaluation and comparison of research performance. A good candidate for evaluating a piece of research is a researcher who cites the research and knows it well. By accumulating different citing authors' evaluations of their references based on a uniform evaluation scheme and synthesizing the evaluations into a single indicator, the qualities of research works, scientists, journals, research groups, and institutions in different disciplines can be assessed and compared. A method consisting of three components is proposed: a reference evaluation scheme, a new reference format, and a new indicator, called the average evaluation intensity. This method combines the advantages of citation count analysis, citation motivation analysis, and peer review, and may help to advance the debate. The potential advantages of and main concerns about the proposed method are discussed. The proposed method may serve as a preliminary theoretical framework that can inspire and advance a quality-oriented approach to the evaluation of research performance. At the current stage, it is best to treat the proposed method as speculation and inspiration rather than as a blueprint for practical implementation.