What Deliberately Degrading Search Quality Tells Us About Discount Functions
Deliberate degradation of search results is a common tool in user experiments. We degrade high-quality search results by inserting non-relevant documents at different ranks. The effect of these manipulations, on a number of commonly-used metrics, is counter-intuitive: the discount functions implicit in P@k, MRR, NDCG, and others do not account for the true relationship between rank and value to the user. We propose an alternative, based on visibility data.
|Collections||ANU Research Publications|
|Source:||Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval|
|01_Thomas_What_Deliberately_Degrading_2011.pdf||73.65 kB||Adobe PDF||Request a copy|
Items in Open Research are protected by copyright, with all rights reserved, unless otherwise indicated.