Assessors agreement: A case study across assessor type, payment levels, query variations and relevance dimensions

Palotti, Joao, Zuccon, Guido, Bernhardt, Johannes, Hanbury, Allan, & Goeuriot, Lorraine (2016) Assessors agreement: A case study across assessor type, payment levels, query variations and relevance dimensions. In International Conference of the Cross-Language Evaluation Forum for European Languages (CLEF 2016), September 5-8, 2016, Évora, Portugal.

[img] Accepted Version (PDF 1MB)
Administrators only until August 2017 | Request a copy from author

View at publisher


Relevance assessments are the cornerstone of Information Retrieval evaluation. Yet, there is only limited understanding of how assessment disagreement influences the reliability of the evaluation in terms of systems rankings. In this paper we examine the role of assessor type (expert vs. layperson), payment levels (paid vs. unpaid), query variations and relevance dimensions (topicality and understandability) and their influence on system evaluation in the presence of disagreements across assessments obtained in the different settings. The analysis is carried out in the context of the CLEF 2015 eHealth Task 2 collection and shows that disagreements between assessors belonging to the same group have little impact on evaluation. It also shows, however, that assessment disagreement found across settings has major impact on evaluation when topical relevance is considered, while it has no impact when understandability assessments are considered.

Impact and interest:

Citation counts are sourced monthly from Scopus and Web of Science® citation databases.

These databases contain citations from different subsets of available publications and different time periods and thus the citation count from each is usually different. Some works are not in either database and no count is displayed. Scopus includes citations from articles published in 1996 onwards, and Web of Science® generally from 1980 onwards.

Citations counts from the Google Scholar™ indexing service can be viewed at the linked Google Scholar™ search.

ID Code: 98863
Item Type: Conference Paper
Refereed: Yes
Keywords: Evaluation; Assessments; Assessors agreement
DOI: 10.1007/978-3-319-44564-9_4
ISBN: 9783319445649
Divisions: Current > Schools > School of Electrical Engineering & Computer Science
Copyright Owner: Copyright 2016 Springer
Deposited On: 15 Sep 2016 22:38
Last Modified: 04 Oct 2016 22:54

Export: EndNote | Dublin Core | BibTeX

Repository Staff Only: item control page