A comprehensive analysis of parameter settings for novelty-biased cumulative gain
Leelanupab, Teerapong, Zuccon, Guido, & Jose, Joemon M. (2012) A comprehensive analysis of parameter settings for novelty-biased cumulative gain. In Chen, Xuewen, Lebanon, Guy, Wang, Haixun, & Zaki, Mohammed (Eds.) Proceedings of the 21st ACM International Conference on Information and Knowledge Management - CIKM '12, ACM Digital Library, Sheraton, Maui, Hawaii, pp. 1950-1954.
In the TREC Web Diversity track, novelty-biased cumulative gain (α-NDCG) is one of the official measures to assess retrieval performance of IR systems. The measure is characterised by a parameter, α, the effect of which has not been thoroughly investigated. We find that common settings of α, i.e. α=0.5, may prevent the measure from behaving as desired when evaluating result diversification. This is because it excessively penalises systems that cover many intents while it rewards those that redundantly cover only few intents. This issue is crucial since it highly influences systems at top ranks. We revisit our previously proposed threshold, suggesting α be set on a query-basis. The intuitiveness of the measure is then studied by examining actual rankings from TREC 09-10 Web track submissions. By varying α according to our query-based threshold, the discriminative power of α-NDCG is not harmed and in fact, our approach improves α-NDCG's robustness. Experimental results show that the threshold for α can turn the measure to be more intuitive than using its common settings.
Impact and interest:
Citation counts are sourced monthly from and citation databases.
These databases contain citations from different subsets of available publications and different time periods and thus the citation count from each is usually different. Some works are not in either database and no count is displayed. Scopus includes citations from articles published in 1996 onwards, and Web of Science® generally from 1980 onwards.
Citations counts from theindexing service can be viewed at the linked Google Scholar™ search.
|Item Type:||Conference Paper|
|Keywords:||Diversity, Evaluation measure, Information storage and retrieval|
|Divisions:||Current > Schools > School of Information Systems
Current > QUT Faculties and Divisions > Science & Engineering Faculty
|Deposited On:||29 May 2014 22:56|
|Last Modified:||11 Jun 2014 01:58|
Repository Staff Only: item control page