Understandability biased evaluation for information retrieval

Zuccon, Guido (2016) Understandability biased evaluation for information retrieval. In Advances in Information Retrieval: 38th European Conference on IR Research, ECIR 2016, Padua, Italy, March 20-23, 2016. Proceedings, Springer International Publishing, Padua, Italy, pp. 280-292.

[img] PDF (348kB)
Administrators only until 10 March 2017 | Request a copy from author

View at publisher

Abstract

Although relevance is known to be a multidimensional concept, information retrieval measures mainly consider one dimension of relevance: topicality. In this paper we propose a method to integrate multiple dimensions of relevance in the evaluation of information retrieval systems. This is done within the gain-discount evaluation framework, which underlies measures like rank-biased precision (RBP), cumulative gain, and expected reciprocal rank. Albeit the proposal is general and applicable to any dimension of relevance, we study specific instantiations of the approach in the context of evaluating retrieval systems with respect to both the topicality and the understandability of retrieved documents. This leads to the formulation of understandability biased evaluation measures based on RBP. We study these measures using both simulated experiments and real human assessments. The findings show that considering both understandability and topicality in the evaluation of retrieval systems leads to claims about system effectiveness that differ from those obtained when considering topicality alone.

Impact and interest:

1 citations in Scopus
Search Google Scholar™

Citation counts are sourced monthly from Scopus and Web of Science® citation databases.

These databases contain citations from different subsets of available publications and different time periods and thus the citation count from each is usually different. Some works are not in either database and no count is displayed. Scopus includes citations from articles published in 1996 onwards, and Web of Science® generally from 1980 onwards.

Citations counts from the Google Scholar™ indexing service can be viewed at the linked Google Scholar™ search.

ID Code: 95668
Item Type: Conference Paper
Refereed: Yes
Additional Information: Volume 9626 of the series Lecture Notes in Computer Science
Additional URLs:
DOI: 10.1007/978-3-319-30671-1_21
ISBN: 9783319306704
Subjects: Australian and New Zealand Standard Research Classification > INFORMATION AND COMPUTING SCIENCES (080000) > LIBRARY AND INFORMATION STUDIES (080700) > Information Retrieval and Web Search (080704)
Divisions: Current > Schools > School of Electrical Engineering & Computer Science
Current > QUT Faculties and Divisions > Science & Engineering Faculty
Copyright Owner: Copyright 2016 Springer International Publishing Switzerland
Copyright Statement: The final publication is available at Springer via http://dx.doi.org/10.1007/978-3-319-30671-1_21
Deposited On: 20 May 2016 00:14
Last Modified: 10 Jun 2016 14:18

Export: EndNote | Dublin Core | BibTeX

Repository Staff Only: item control page