lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doron Cohen <>
Subject Re: search quality - assessment & improvements
Date Fri, 29 Jun 2007 09:43:43 GMT
Nadav Har'El wrote:

> Another approach is to use Term Relevance Sets, described in [1].
> This new approach not only requires less manual labor than
> TREC's approach,
> but also works better when the corpus is evolving.
> [1] "Scaling IR-System Evaluation using Term Relevance Sets",
> Einat Amitay,
> David Carmel, Ronny Lempel and Aya Soffer, SIGIR 2004,

This is an interesting approach; It says (more or less): creating TREC's
QRels is based on human labor, judging documents of the searched collection
against the "topics" (queries). Since there are too many docs (and in some
Tracks too many queries) sampling techniques are used and only some
docs/queries are manually judged. In the proposed TRel approach an
(original) query is decorated with lists of positive and negative terms.
Then, returned docs are evaluated against these term lists: the existence
of positive terms in the returned doc increases its "relevant match
probability", and the existence of a negative term reduces that.

Well in general I like this idea, and the paper shows correlation between
TRel judgments and QRel judgements. But I have a problem with the judging
procedure using the same techniques being evaluated - checking term
frequencies that is. Also, one could devise queries that are actually made
of the "plus terms" and "minus terms"... I find the "magical" attempt to
have your automated system find what a human would have selected, (was he
able to read all docs), more, correct, appealing...

Anyhow my take from this, at least, is that the quality checking mechanism
that we want to add to the benchmark should be general/open enough to allow
this notion of quality assessing, or any other notion.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message