A comprehensive analysis of parameter settings for novelty-biased cumulative gain

Leelanupab, Teerapong, Zuccon, Guido, & Jose, Joemon M. (2012) A comprehensive analysis of parameter settings for novelty-biased cumulative gain. In Chen, Xuewen, Lebanon, Guy, Wang, Haixun, & Zaki, Mohammed (Eds.) Proceedings of the 21st ACM International Conference on Information and Knowledge Management - CIKM '12, ACM Digital Library, Sheraton, Maui, Hawaii, pp. 1950-1954.

View at publisher

Abstract

In the TREC Web Diversity track, novelty-biased cumulative gain (α-NDCG) is one of the official measures to assess retrieval performance of IR systems. The measure is characterised by a parameter, α, the effect of which has not been thoroughly investigated. We find that common settings of α, i.e. α=0.5, may prevent the measure from behaving as desired when evaluating result diversification. This is because it excessively penalises systems that cover many intents while it rewards those that redundantly cover only few intents. This issue is crucial since it highly influences systems at top ranks. We revisit our previously proposed threshold, suggesting α be set on a query-basis. The intuitiveness of the measure is then studied by examining actual rankings from TREC 09-10 Web track submissions. By varying α according to our query-based threshold, the discriminative power of α-NDCG is not harmed and in fact, our approach improves α-NDCG's robustness. Experimental results show that the threshold for α can turn the measure to be more intuitive than using its common settings.

Impact and interest:

3 citations in Scopus
Search Google Scholar™

Citation counts are sourced monthly from Scopus and Web of Science® citation databases.

These databases contain citations from different subsets of available publications and different time periods and thus the citation count from each is usually different. Some works are not in either database and no count is displayed. Scopus includes citations from articles published in 1996 onwards, and Web of Science® generally from 1980 onwards.

Citations counts from the Google Scholar™ indexing service can be viewed at the linked Google Scholar™ search.

ID Code: 72216
Item Type: Conference Paper
Refereed: Yes
Additional URLs:
Keywords: Diversity, Evaluation measure, Information storage and retrieval
DOI: 10.1145/2396761.2398550
ISBN: 9781450311564
Divisions: Current > Schools > School of Information Systems
Current > QUT Faculties and Divisions > Science & Engineering Faculty
Deposited On: 29 May 2014 22:56
Last Modified: 11 Jun 2014 01:58

Export: EndNote | Dublin Core | BibTeX

Repository Staff Only: item control page