lucene-openrelevance-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Grant Ingersoll <>
Subject Fwd: [SIG-IRList] Call for papers: NGTC - The SIGIR 2010 Workshop on Next Generation Test Collections
Date Mon, 03 May 2010 20:59:42 GMT

Begin forwarded message:

> From: Ian Soboroff <>
> Date: April 29, 2010 3:26:27 PM EDT
> To:
> Subject: [SIG-IRList] Call for papers: NGTC - The SIGIR 2010 Workshop on Next Generation
Test Collections
> Next-Generation Test Collections (NGTC '10)
> Call for Papers
> We solicit submissions for the workshop on Next-Generation Test Collections, to be held
on July 23, 2010, in Geneva, Switzerland, in conjunction with the 33rd Annual International
ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2010). The
workshop will bring together researchers and practitioners from academia and industry to discuss
future developments of test collections for textual information retrieval.
> Motivation
> In 2003, SIGIR held a workshop on evaluation methodologies for terabyte-scale test collections.
At that time, we were trying to anticipate the kinds of difficulties that would arise from
rapid upscaling of the Cranfield paradigm, and from that discussion grew several lines of
research into measures, sampling methodologies, and evaluation frameworks.
> In the intervening years, TREC document collections have increased in size from gigabytes
to hundreds of gigabytes, and now tens of terabytes. We are beginning to see that our ideas
about the challenges of terabyte-scale (and larger!) test collections were not quite on the
mark. Research has shown that current very large TREC collections suffer from judgment-set
bias and reusability challenges more complex than our early expectations envisioned.
> This workshop invites cutting-edge research on tackling the problem of building test
collections at the multi-terabyte scale that are realistic, fair, and reusable. The goal of
the workshop is to map out the critical research questions that need to be asked and the types
of collections we need to consider building in order to answer them.
> Topics
> Appropriate subjects for papers include, but are not limited to
> methods for acquiring relevance judgments
> new approaches and modified approaches
> analysis of the scalability of approaches to judgment collection
> comparing the ability of approaches to produce realistic and reusable test collections
> measures of retrieval effectiveness
> new measures that adjust for bias or limited coverage in judgments
> analysis of the robustness of existing measures to bias or limited coverage
> evaluation of the applicability of measures to tasks
> evaluation of novelty, diversity, churn, recency
> measures of test collection quality and reusability
> analysis of representativeness of a test collection for a task
> analysis of the reusability of a test collection to train and evaluate new systems
> methods for improving reusability or test collection quality
> using test collections designed for one task to experiment with other tasks
> task definition and topic set selection
> the effect of task definition and topic set on the quality and reusability of a test
> methods for selecting topic sets for test collections
> approaches to careful task definition
> run-gathering strategies
> analysis of the diversity of runs needed to ensure high-quality, reusable collections
> methods for sampling runs, estimating priors over runs, or generating quasi-runs to increase
> Submission Instructions
> Research contributions should report new unpublished research results, research work
in progress, or positions, ideas, and challenges for test collection design. Submissions can
be full papers up to eight pages long (for oral presentation) or short papers up to two pages
long (for poster presentation) and should be formatted in double-column ACM SIG proceedings
> Papers should be submitted electronically using the conference management system:

> Important Dates
> June 4, 2010	Submission deadline
> June 18, 2010	Notification of acceptance
> July 2, 2010	Camera-ready submission
> July 23, 2010	Workshop in Geneva
> Organizing Committee
> Ian Soboroff, NIST 
> Ben Carterette, University of Delaware 
> Virgil Pavlu, Northeastern University
> Program Committee
> Jay Aslam, Northeastern University
> Peter Bailey, Microsoft
> Charlie Clarke, University of Waterloo
> Evgeniy Gabrilovitch, Yahoo!
> Scott Huffman, Google
> Evangelos Kanoulas, University of Sheffield
> Don Metzler, ISI, USC
> Jan Pedersen, Microsoft
> Mark Sanderson, University of Sheffield
> Ellen Voorhees, NIST
> William Webber, University of Melbourne
> ************************************************ 
> This SIGIR-IRList message and the SIG-IRList Digest (a moderated IR newsletter), are
brought to you by SIGIR, distributed from the University of Sheffield and edited by Mark Smucker
> o	To submit an article, e-mail 
> o	To subscribe, send mail to , with the subject: SUBSCRIBE irlist
firstname lastname
> o	To unsubscribe, send mail to, with the subject: UNSUBSCRIBE
irlist email
> [The email address is required only if you want to unsubscribe with an address other
than the address with which you send the message]
> o	For more info, visit:
> These files are not to be sold or used for commercial purposes. 

Grant Ingersoll

Search the Lucene ecosystem using Solr/Lucene:

View raw message