Skip navigation
Skip navigation

What Deliberately Degrading Search Quality Tells Us About Discount Functions

Thomas, Paul; Jones, Timothy; Hawking, David


Deliberate degradation of search results is a common tool in user experiments. We degrade high-quality search results by inserting non-relevant documents at different ranks. The effect of these manipulations, on a number of commonly-used metrics, is counter-intuitive: the discount functions implicit in P@k, MRR, NDCG, and others do not account for the true relationship between rank and value to the user. We propose an alternative, based on visibility data.

CollectionsANU Research Publications
Date published: 2011
Type: Conference paper
Source: Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval
DOI: 10.1145/2009916.2010072


File Description SizeFormat Image
01_Thomas_What_Deliberately_Degrading_2011.pdf73.65 kBAdobe PDF    Request a copy

Items in Open Research are protected by copyright, with all rights reserved, unless otherwise indicated.

Updated:  20 July 2017/ Responsible Officer:  University Librarian/ Page Contact:  Library Systems & Web Coordinator