A little competition never hurt anyone's relevance assessments

Abstract

This paper investigates the effect of real-time performance feedback and competition on the accuracy of crowd-workers for a document relevance assessment task. Through a series of controlled tests, we show that displaying a leaderboard to crowd-workers can motivate them to improve their performance, providing a bonus is offered to the best performing workers. This effect is observed even when test questions are used to enforce quality control during task completion. Show more