lucene-openrelevance-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicola Ferro <>
Subject CFP CIKM 2011 Workshop on Data infrastructurEs for Supporting Information Retrieval Evaluation - Extended Deadline 9 July 2011
Date Tue, 21 Jun 2011 15:52:18 GMT
** Apologies for cross postings **

Call for Papers - Extended Deadline July 9, 2011

Data infrastructurEs for Supporting Information Retrieval Evaluation (DESIRE 2011) Workshop

Co-located with CIKM 2011, the 20th ACM Conference on Information and Knowledge Management

Friday 28th October 2011, Glasgow, UK


The Information Retrieval area has a strong and long tradition dating back to the 1960s in
and processing scientific data resulting from the experimental evaluation of search algorithms
search systems. This attitude towards evaluation has led to fast and continuous progress in
evolution of information retrieval systems and search engines. 

However, in order to make these data test collections understandable and usable they must
be endowed 
with some auxiliary information, i.e., provenance, quality, context, etc. Therefore, there
is a need 
for metadata models able to describe the main characteristics of evaluation data. In addition,
order to make distributed data collections accessible, sharable, and interoperable, there
is a need 
for advanced data infrastructures.

Nevertheless, the information retrieval area has barely explored and exploited the possibilities
managing, storing, and effectively accessing the scientific data produced during the evaluation

studies by making use of the methods typical of the database and knowledge management areas.

It is thus time for these three communities – information retrieval, databases, and 
knowledge management – to join efforts, meet, and cooperate to address the problem of envisaging
designing useful infrastructures able to coherently manage pertinent data collections and
of information, and so take concrete steps towards developing them.

Indeed, the information retrieval experts need to recognise this need, while the database
knowledge management experts need to understand the problem and work together to solve it
by using 
the methods and techniques specific to information management.

Therefore, the workshop will gather together experts from these three areas,  to encourage
them to 
recognise the urgency of addressing the problem in an integrated and coherent way, and to
efforts towards drawing a roadmap and suggesting best practices for an effective solution
of the 

The objectives of the workshop are: to gather interested experts from the three areas of interest

to CIKM and discuss how to address the problem of envisaging and designing evaluation 
infrastructures able to store, manage, and make accessible the scientific data and knowledge
interest for advancing the evaluation of information retrieval and access tools.

The goal of the workshop is to understand how to make use of the expertise of the three scientific

areas in a cooperative way to avoid the duplication of efforts which may occur when addressing
problem separately in each specific area and to trigger synergies and joint actions on this

The workshop is intended to be a single full day workshop, consisting of: invited talks and
presentations during the morning; active discussion groups during the afternoon, engaging
participants in the topics which have emerged in the previous sessions in order to define
a roadmap 
and best practices.

The main outcome will be to produce a roadmap and initial best practices to guide the development

of the necessary evaluation infrastructures. 


- Conceptual and logical data models for representing IR evaluation scientific data
- Metadata formats for describing scientific data produced during IR evaluation
- Knowledge management for IR experimental evaluation
- Data quality, provenance, adaptability and reusability in the IR evaluation
- Data pre- and post-processing, metrics, and analyses in the IR evaluation
- Data exchange, integration, evolution and migration for IR evaluation infrastructures
- Workflow, Web services and Web Service Composition for IR evaluation infrastructures
- Information Extraction and Text Mining for linking scientific literature and experimental
- Data citation
- Evaluation, Test collections, Crowdsourcing for IR evaluation
- Visualization of scientific data coming from experimental evaluation


- Individual Workshop Papers Due: July 9, 2011 (extended)
- Notification of Acceptance: July 18, 2011
- Camera Ready: July 22, 2011


Authors are invited to submit electronically original papers, which have not been published
are not under consideration elsewhere, using the ACM template Option 2. including the ACM
(General Terms, Categories, Keywords, Copyright number etc), available at:

Two types of comunications or position papers are solicited:

- communications: 4 pages max.
- position papers: 6 pages max.

Papers will be peer-reviewed by at least 3 members of the program committee. Selection will
based on originality, clarity, and technical quality. 

Papers should be submitted in PDF format to the following address:


- Norbert Fuhr, University of Duisburg-Essen, Germany.
- Maurizio Lenzerini, Sapienza University of Rome, Italy.


- Stefano Ceri, Politecnico di Milano, Italy
- Maarten de Rijke, University of Amsterdam, The Netherlands
- Donna Harman, National Institute of Standards and Technology (NIST), USA
- Noriko Kando, National Institute of Informatics (NII), Japan
- Henning Müller, University of Applied Sciences Western Switzerland
- Shila Ofek-Koifman, IBM Haifa, Israel
- Christos Papatheodorou, Ionian University, Greece
- Fausto Rabitti, ISTI-CNR, Italy
- Mark Sanderson, RMIT University, Australia
- Timos Sellis, National Technical University of Athens, Greece
- Gerhard Weikum, Max-Planck-Institut für Informatik, Germany


- Maristella Agosti, University of Padua, Italy
- Nicola Ferro, University of Padua, Italy
- Costantino Thanos, ISTI-CNR, National Council of Research, Italy


- Nicola Ferro, ferro@dei.unipd.It

View raw message