Thomas, PaulJones, TimothyHawking, David2015-12-07July 24-289781450309349http://hdl.handle.net/1885/19198Deliberate degradation of search results is a common tool in user experiments. We degrade high-quality search results by inserting non-relevant documents at different ranks. The effect of these manipulations, on a number of commonly-used metrics, is counter-intuitive: the discount functions implicit in P@k, MRR, NDCG, and others do not account for the true relationship between rank and value to the user. We propose an alternative, based on visibility data.Keywords: High quality; Metrics; Non-relevant documents; Result set manipulation; Search quality; Search results; Information retrieval Metrics; Result set manipulation201110.1145/2009916.20100722016-02-24