The impact of fixed-cost pooling strategies on test collection bias

Lipani, Aldo, Zuccon, Guido, Lupu, Mihai, Koopman, Bevan, & Hanbury, Allan (2016) The impact of fixed-cost pooling strategies on test collection bias. In ACM International Conference on the Theory of Information Retrieval (ICTIR '16), September 12 - 16, 2016, Newark, Delaware.

View at publisher


In Information Retrieval, test collections are usually built using the pooling method. Many pooling strategies have been developed for the pooling method. Herein, we address the question of identifying the best pooling strategy when evaluating systems using precision-oriented measures in presence of budget constraints on the number of documents to be evaluated. As a quality measurement we use the bias introduced by the pooling strategy, measured both in terms of Mean Absolute Error of the scores and in terms of ranking errors. Based on experiments on 15 test collections, we conclude that, for precision-oriented measures, the best strategies are based on Rank-Biased Precision (RBP). These results can inform collection builders because they suggest that, under fixed assessment budget constraints, RBP-based sampling produces less biased pools than other alternatives.

Impact and interest:

1 citations in Scopus
Search Google Scholar™

Citation counts are sourced monthly from Scopus and Web of Science® citation databases.

These databases contain citations from different subsets of available publications and different time periods and thus the citation count from each is usually different. Some works are not in either database and no count is displayed. Scopus includes citations from articles published in 1996 onwards, and Web of Science® generally from 1980 onwards.

Citations counts from the Google Scholar™ indexing service can be viewed at the linked Google Scholar™ search.

ID Code: 98867
Item Type: Conference Paper
Refereed: Yes
Additional URLs:
Keywords: Pooling Method, Pooling Strategies, Pool Bias
DOI: 10.1145/2970398.2970429
ISBN: 9781450344975
Divisions: Current > Schools > School of Electrical Engineering & Computer Science
Copyright Owner: Copyright held by the owner/author(s). Publication rights licensed to ACM.
Deposited On: 15 Sep 2016 22:38
Last Modified: 14 Mar 2017 03:49

Export: EndNote | Dublin Core | BibTeX

Repository Staff Only: item control page