๐Ÿ“–
Overview of the INEX 2010 Book Track: Scaling Up the Evaluation Using Crowdsourcing
Gabriella Kazai, Marijn Koolen, Jaap Kamps, Antoine Doucet, Monica Landoni

Overview of the INEX 2010 Book Track: Scaling Up the Evaluation Using Crowdsourcing

Science & Mathematics Published 2011 Verification Pending

Overview & Synopsis

The goal of the INEX Book Track is to evaluate approaches for supporting users in searching, navigating and reading the full texts of digitized books. The investigation is focused around four tasks: 1) Best Books to Reference, 2) Prove It, 3) Structure Extraction, and 4) Active Reading. In this paper, we report on the setup and the results of these tasks in 2010. The main outcome of the track lies in the changes to the methodology for constructing the test collection for the evaluation of the Best Books and Prove It search tasks. In an effort to scale up the evaluation, we explored the use of crowdsourcing both to create the test topics and then to gather the relevance labels for the topics over a corpus of 50k digitized books. The resulting test collection construction methodology combines editorial judgments contributed by INEX participants with crowdsourced relevance labels. We provide an analysis of the crowdsourced data and conclude that - with appropriate task design - crowdsourcing does provide a suitable framework for the evaluation of book search approaches.

Indexed from repository: HAL Open Science (hal.science)
Loading document from verified source...
Keyboard Shortcuts: ← / → for pages • + / - for zoom • 0 to fit width Powered by Mozilla PDF.js v3.11

Document Information

Document Title Overview of the INEX 2010 Book Track: Scaling Up the Evaluation Using Crowdsourcing
Author / Organization Gabriella Kazai, Marijn Koolen, Jaap Kamps, Antoine Doucet, Monica Landoni
Subject & Discipline Multidisciplinary Academic Open Research
Categories Science & Mathematics
Publication Year 2011
License / Rights Open Access / Creative Commons
Resource Verification Listed PDF (Verification Pending)
Original Host Website HAL Open Science (hal.science)
File Format application/pdf (Portable Document Format)
Language EN
Catalog ID DOC-156699
Original Source Link Visit Source Page on hal.science »