lucene-openrelevance-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ludovico Boratto <ludovico.bora...@gmail.com>
Subject Re: Calculating a search engine's MAP
Date Wed, 16 Dec 2009 13:26:18 GMT
Hi,
thanks for your reply.
How can trec_eval work properly?
A standard TREC rank contains 1000 results, while the relevants judgments
are about a much smaller amount of documents (usually 50).

How can I calculate precision and recall if I don't know how relevant are
the 95% of the documents in the ranking I produced?

Thanks in advance for your help,
Ludovico

2009/12/9 Grant Ingersoll <gsingers@apache.org>

>
> On Dec 9, 2009, at 9:46 AM, Ludovico Boratto wrote:
>
> > Hi everyone,
> > I'm a PhD student, and I was wondering how is it possible to evaluate a
> search engine performances with a dataset like the ones made available for
> the TREC tracks.
> >
> > The problem I have is: once I submit a query and the search engine forms
> a list of ranked results, how do I know which documents are relevant and the
> ones that are not?
> > I know that with each track there are relevant judgments available, but
> those judgments are about a small number of queries and documents.
> >
>
> I think there is two questions in here, if I'm not mistaken.  First, TREC
> delivers a set of qrels files that capture this information based on the way
> they do relevance pooling.  If you search for trec_eval, you will find a
> tool that takes in your results and the TREC judgments and outputs MAP, etc.
>

The second question I'm inferring is you want to know how does this apply to
> your search application.  That is, how do you judge relevance for your app.
>   The answer to that is a bit harder.  Essentially, you need to go through
> and create the queries and judgments.  Many people use log analysis to
> achieve this.  You might find
> http://www.lucidimagination.com/Community/Hear-from-the-Experts/Articles/Debugging-Relevance-Issues-Searchhelpful.
>
>
> > Since these are my first steps in the IR world, I hope you don't mind
> helping me, please.
> >
> > Thanks in advance for your help, I'm looking forward to hearing from you
> soon.
> > Yours faithfully,
> > Ludovico
>
> --------------------------
> Grant Ingersoll
> http://www.lucidimagination.com/
>
> Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids) using
> Solr/Lucene:
> http://www.lucidimagination.com/search
>
>

Mime
View raw message