Is intent-aware expected reciprocal rank sufficient to evaluate diversity?

Leelanupab, Teerapong, Zuccon, Guido, & Jose, Joemon M. (2013) Is intent-aware expected reciprocal rank sufficient to evaluate diversity? In Serdyukov, Pavel, Braslavski, Pavel, Kuznetsov, Sergei O., Kamps, Jaap, Rüger, Stefan, Agichtein, Eugene, et al. (Eds.) Advances in Information Retrieval. Springer Berlin Heidelberg, pp. 738-742.

View at publisher


In this paper we define two models of users that require diversity in search results; these models are theoretically grounded in the notion of intrinsic and extrinsic diversity. We then examine Intent-Aware Expected Reciprocal Rank (ERR-IA), one of the official measures used to assess diversity in TREC 2011-12, with respect to the proposed user models. By analyzing ranking preferences as expressed by the user models and those estimated by ERR-IA, we investigate whether ERR-IA assesses document rankings according to the requirements of the diversity retrieval task expressed by the two models. Empirical results demonstrate that ERR-IA neglects query-intents coverage by attributing excessive importance to redundant relevant documents. ERR-IA behavior is contrary to the user models that require measures to first assess diversity through the coverage of intents, and then assess the redundancy of relevant intents. Furthermore, diversity should be considered separately from document relevance and the documents positions in the ranking.

Impact and interest:

1 citations in Scopus
Search Google Scholar™

Citation counts are sourced monthly from Scopus and Web of Science® citation databases.

These databases contain citations from different subsets of available publications and different time periods and thus the citation count from each is usually different. Some works are not in either database and no count is displayed. Scopus includes citations from articles published in 1996 onwards, and Web of Science® generally from 1980 onwards.

Citations counts from the Google Scholar™ indexing service can be viewed at the linked Google Scholar™ search.

ID Code: 69303
Item Type: Book Chapter
Additional Information: Book Subtitle:
35th European Conference on IR Research, ECIR 2013, Moscow, Russia, March 24-27, 2013. Proceedings
DOI: 10.1007/978-3-642-36973-5_73
ISBN: 9783642369735
Divisions: Current > Institutes > Institute for Future Environments
Current > Schools > School of Information Systems
Current > QUT Faculties and Divisions > Science & Engineering Faculty
Copyright Owner: Copyright 2013 Springer-Verlag Berlin Heidelberg
Deposited On: 07 May 2014 22:33
Last Modified: 25 Oct 2015 16:24

Export: EndNote | Dublin Core | BibTeX

Repository Staff Only: item control page