Is intent-aware expected reciprocal rank sufficient to evaluate diversity?
Leelanupab, Teerapong, Zuccon, Guido, & Jose, Joemon M. (2013) Is intent-aware expected reciprocal rank sufficient to evaluate diversity? In Serdyukov, Pavel, Braslavski, Pavel, Kuznetsov, Sergei O., Kamps, Jaap, Rüger, Stefan, Agichtein, Eugene, et al. (Eds.) Advances in Information Retrieval. Springer Berlin Heidelberg, pp. 738-742.
In this paper we define two models of users that require diversity in search results; these models are theoretically grounded in the notion of intrinsic and extrinsic diversity. We then examine Intent-Aware Expected Reciprocal Rank (ERR-IA), one of the official measures used to assess diversity in TREC 2011-12, with respect to the proposed user models. By analyzing ranking preferences as expressed by the user models and those estimated by ERR-IA, we investigate whether ERR-IA assesses document rankings according to the requirements of the diversity retrieval task expressed by the two models. Empirical results demonstrate that ERR-IA neglects query-intents coverage by attributing excessive importance to redundant relevant documents. ERR-IA behavior is contrary to the user models that require measures to first assess diversity through the coverage of intents, and then assess the redundancy of relevant intents. Furthermore, diversity should be considered separately from document relevance and the documents positions in the ranking.
Impact and interest:
Citation counts are sourced monthly from and citation databases.
These databases contain citations from different subsets of available publications and different time periods and thus the citation count from each is usually different. Some works are not in either database and no count is displayed. Scopus includes citations from articles published in 1996 onwards, and Web of Science® generally from 1980 onwards.
Citations counts from theindexing service can be viewed at the linked Google Scholar™ search.
|Item Type:||Book Chapter|
|Additional Information:||Book Subtitle:
35th European Conference on IR Research, ECIR 2013, Moscow, Russia, March 24-27, 2013. Proceedings
|Divisions:||Current > Institutes > Institute for Future Environments
Current > Schools > School of Information Systems
Current > QUT Faculties and Divisions > Science & Engineering Faculty
|Copyright Owner:||Copyright 2013 Springer-Verlag Berlin Heidelberg|
|Deposited On:||07 May 2014 22:33|
|Last Modified:||25 Oct 2015 16:24|
Repository Staff Only: item control page