Overview of the INEX 2010 focused relevance feedback track

Abstract

The INEX 2010 Focused Relevance Feedback track offered
a refined approach to the evaluation of Focused Relevance Feedback algorithms through simulated exhaustive user feedback. As in traditional approaches we simulated a user-in-the loop by re-using the assessments of ad-hoc retrieval obtained from real users who assess focused ad-hoc
retrieval submissions. The evaluation was extended in several ways: the use of exhaustive relevance feedback over entire runs; the evaluation of focused retrieval where
both the retrieval results and the feedback are focused; the evaluation was performed over a closed set of documents and complete focused assessments; the evaluation was performed over executable implementations of relevance feedback algorithms; and �finally, the entire evaluation
platform is reusable. We present the evaluation methodology, its implementation, and experimental results obtained for nine submissions from three participating
organisations.