From: Ian Soboroff <firstname.lastname@example.org>Date: April 29, 2010 3:26:27 PM EDTSubject: [SIG-IRList] Call for papers: NGTC - The SIGIR 2010 Workshop on Next Generation Test Collections
Next-Generation Test Collections (NGTC '10)http://ir.cis.udel.edu/NGTC10/cfp.html
Call for PapersWe solicit submissions for the workshop on Next-Generation Test Collections, to be held on July 23, 2010, in Geneva, Switzerland, in conjunction with the 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2010). The workshop will bring together researchers and practitioners from academia and industry to discuss future developments of test collections for textual information retrieval.
MotivationIn 2003, SIGIR held a workshop on evaluation methodologies for terabyte-scale test collections. At that time, we were trying to anticipate the kinds of difficulties that would arise from rapid upscaling of the Cranfield paradigm, and from that discussion grew several lines of research into measures, sampling methodologies, and evaluation frameworks.
In the intervening years, TREC document collections have increased in size from gigabytes to hundreds of gigabytes, and now tens of terabytes. We are beginning to see that our ideas about the challenges of terabyte-scale (and larger!) test collections were not quite on the mark. Research has shown that current very large TREC collections suffer from judgment-set bias and reusability challenges more complex than our early expectations envisioned.
This workshop invites cutting-edge research on tackling the problem of building test collections at the multi-terabyte scale that are realistic, fair, and reusable. The goal of the workshop is to map out the critical research questions that need to be asked and the types of collections we need to consider building in order to answer them.
TopicsAppropriate subjects for papers include, but are not limited to
- methods for acquiring relevance judgments
- new approaches and modified approaches
- analysis of the scalability of approaches to judgment collection
- comparing the ability of approaches to produce realistic and reusable test collections
- measures of retrieval effectiveness
- new measures that adjust for bias or limited coverage in judgments
- analysis of the robustness of existing measures to bias or limited coverage
- evaluation of the applicability of measures to tasks
- evaluation of novelty, diversity, churn, recency
- measures of test collection quality and reusability
- analysis of representativeness of a test collection for a task
- analysis of the reusability of a test collection to train and evaluate new systems
- methods for improving reusability or test collection quality
- using test collections designed for one task to experiment with other tasks
- task definition and topic set selection
- the effect of task definition and topic set on the quality and reusability of a test collection
- methods for selecting topic sets for test collections
- approaches to careful task definition
- run-gathering strategies
- analysis of the diversity of runs needed to ensure high-quality, reusable collections
- methods for sampling runs, estimating priors over runs, or generating quasi-runs to increase diversity
Submission InstructionsResearch contributions should report new unpublished research results, research work in progress, or positions, ideas, and challenges for test collection design. Submissions can be full papers up to eight pages long (for oral presentation) or short papers up to two pages long (for poster presentation) and should be formatted in double-column ACM SIG proceedings format.
Papers should be submitted electronically using the conference management system: http://www.easychair.org/conferences/?conf=ngtc10.
June 4, 2010 Submission deadline June 18, 2010 Notification of acceptance July 2, 2010 Camera-ready submission July 23, 2010 Workshop in Geneva
Organizing CommitteeIan Soboroff, NIST
Ben Carterette, University of Delaware
Virgil Pavlu, Northeastern University
Program CommitteeJay Aslam, Northeastern University
Peter Bailey, Microsoft
Charlie Clarke, University of Waterloo
Evgeniy Gabrilovitch, Yahoo!
Scott Huffman, Google
Evangelos Kanoulas, University of Sheffield
Don Metzler, ISI, USC
Jan Pedersen, Microsoft
Mark Sanderson, University of Sheffield
Ellen Voorhees, NIST
William Webber, University of Melbourne
This SIGIR-IRList message and the SIG-IRList Digest (a moderated IR newsletter), are brought to you by SIGIR, distributed from the University of Sheffield and edited by Mark Smucker (email@example.com).
o To submit an article, e-mail IRList@lists.shef.ac.uk
o To subscribe, send mail to firstname.lastname@example.org , with the subject: SUBSCRIBE irlist firstname lastname
o To unsubscribe, send mail to email@example.com, with the subject: UNSUBSCRIBE irlist email
[The email address is required only if you want to unsubscribe with an address other than the address with which you send the message]
o For more info, visit: http://www.sigir.org/sigirlist/
These files are not to be sold or used for commercial purposes.
THE OPINIONS EXPRESSED WITHIN THIS DOCUMENT DO NOT REPRESENT THOSE OF THE EDITOR, THE UNIVERSITY OF WATERLOO OR THE UNIVERSITY OF SHEFFIELD.
AUTHORS ASSUME FULL RESPONSIBILITY FOR THEIR MATERIAL.