Crowdsourcing interactions : using crowdsourcing for evaluating interactive information retrieval systems

Zuccon, Guido, Leelanupab, Teerapong, Whiting, Stewart, Yilmaz, Emine, Jose, Joemon M., & Azzopardi, Leif (2012) Crowdsourcing interactions : using crowdsourcing for evaluating interactive information retrieval systems. Information Retrieval, 16(2), pp. 267-305.

View at publisher


In the field of information retrieval (IR), researchers and practitioners are often faced with a demand for valid approaches to evaluate the performance of retrieval systems. The Cranfield experiment paradigm has been dominant for the in-vitro evaluation of IR systems. Alternative to this paradigm, laboratory-based user studies have been widely used to evaluate interactive information retrieval (IIR) systems, and at the same time investigate users’ information searching behaviours. Major drawbacks of laboratory-based user studies for evaluating IIR systems include the high monetary and temporal costs involved in setting up and running those experiments, the lack of heterogeneity amongst the user population and the limited scale of the experiments, which usually involve a relatively restricted set of users. In this paper, we propose an alternative experimental methodology to laboratory-based user studies. Our novel experimental methodology uses a crowdsourcing platform as a means of engaging study participants. Through crowdsourcing, our experimental methodology can capture user interactions and searching behaviours at a lower cost, with more data, and within a shorter period than traditional laboratory-based user studies, and therefore can be used to assess the performances of IIR systems. In this article, we show the characteristic differences of our approach with respect to traditional IIR experimental and evaluation procedures. We also perform a use case study comparing crowdsourcing-based evaluation with laboratory-based evaluation of IIR systems, which can serve as a tutorial for setting up crowdsourcing-based IIR evaluations.

Impact and interest:

8 citations in Scopus
Search Google Scholar™
3 citations in Web of Science®

Citation counts are sourced monthly from Scopus and Web of Science® citation databases.

These databases contain citations from different subsets of available publications and different time periods and thus the citation count from each is usually different. Some works are not in either database and no count is displayed. Scopus includes citations from articles published in 1996 onwards, and Web of Science® generally from 1980 onwards.

Citations counts from the Google Scholar™ indexing service can be viewed at the linked Google Scholar™ search.

ID Code: 70149
Item Type: Journal Article
Refereed: Yes
Keywords: Crowdsourcing evaluation, Interactive IR evaluation, Interactions
DOI: 10.1007/s10791-012-9206-z
ISSN: 1573-7659
Divisions: Current > Schools > School of Information Systems
Current > QUT Faculties and Divisions > Science & Engineering Faculty
Deposited On: 14 Apr 2014 03:15
Last Modified: 15 Apr 2014 01:57

Export: EndNote | Dublin Core | BibTeX

Repository Staff Only: item control page