Overview of the CLEF eHealth Evaluation Lab 2015

Goeuriot, Lorraine, Kelly, Liadh, Suominen, Hanna, Hanlen, Leif, Névéol, Aurélie, Grouin, Cyril, Palotti, Joao, & Zuccon, Guido (2015) Overview of the CLEF eHealth Evaluation Lab 2015. In Experimental IR Meets Multilinguality, Multimodality, and Interaction: 6th International Conference of the CLEF Association, CLEF'15, Toulouse, France, September 8-11, 2015, Proceeding, Springer International Publishing, Toulouse, France, pp. 429-443.

[img] PDF (401kB)
Administrators only until 20 November 2016 | Request a copy from author

View at publisher


This paper reports on the 3rd CLEFeHealth evaluation lab, which continues our evaluation resource building activities for the medical domain. In this edition of the lab, we focus on easing patients and nurses in authoring, understanding, and accessing eHealth information. The 2015 CLEFeHealth evaluation lab was structured into two tasks, focusing on evaluating methods for information extraction (IE) and information retrieval (IR). The IE task introduced two new challenges. Task 1a focused on clinical speech recognition of nursing handover notes; Task 1b focused on clinical named entity recognition in languages other than English, specifically French. Task 2 focused on the retrieval of health information to answer queries issued by general consumers seeking information to understand their health symptoms or conditions.

The number of teams registering their interest was 47 in Tasks 1 (2 teams in Task 1a and 7 teams in Task 1b) and 53 in Task 2 (12 teams) for a total of 20 unique teams. The best system recognized 4, 984 out of 6, 818 test words correctly and generated 2, 626 incorrect words (i.e., 38.5% error) in Task 1a; had the F-measure of 0.756 for plain entity recognition, 0.711 for normalized entity recognition, and 0.872 for entity normalization in Task 1b; and resulted in P@10 of 0.5394 and nDCG@10 of 0.5086 in Task 2. These results demonstrate the substantial community interest and capabilities of these systems in addressing challenges faced by patients and nurses. As in previous years, the organizers have made data and tools available for future research and development.

Impact and interest:

0 citations in Scopus
Search Google Scholar™

Citation counts are sourced monthly from Scopus and Web of Science® citation databases.

These databases contain citations from different subsets of available publications and different time periods and thus the citation count from each is usually different. Some works are not in either database and no count is displayed. Scopus includes citations from articles published in 1996 onwards, and Web of Science® generally from 1980 onwards.

Citations counts from the Google Scholar™ indexing service can be viewed at the linked Google Scholar™ search.

ID Code: 95676
Item Type: Conference Paper
Refereed: Yes
Additional Information: Volume 9283 of the series Lecture Notes in Computer Science
Additional URLs:
Keywords: Evaluation, Information retrieval, Information extraction, Medical informatics, Nursing records, Patient handoff/handover, Speech recognition, Test-set generation, Text classification, Text segmentation, Self-diagnosis
DOI: 10.1007/978-3-319-24027-5_44
ISBN: 9783319240268
Subjects: Australian and New Zealand Standard Research Classification > INFORMATION AND COMPUTING SCIENCES (080000) > LIBRARY AND INFORMATION STUDIES (080700) > Health Informatics (080702)
Australian and New Zealand Standard Research Classification > INFORMATION AND COMPUTING SCIENCES (080000) > LIBRARY AND INFORMATION STUDIES (080700) > Information Retrieval and Web Search (080704)
Divisions: Current > Schools > School of Electrical Engineering & Computer Science
Current > QUT Faculties and Divisions > Science & Engineering Faculty
Copyright Owner: Copyright 2015 Springer International Publishing Switzerland
Copyright Statement: The final publication is available at Springer via http://dx.doi.org/10.1007/978-3-319-24027-5_44
Deposited On: 20 May 2016 02:59
Last Modified: 10 Jun 2016 09:44

Export: EndNote | Dublin Core | BibTeX

Repository Staff Only: item control page