lucene-openrelevance-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Grant Ingersoll <gsing...@apache.org>
Subject Re: Calculating a search engine's MAP
Date Wed, 09 Dec 2009 17:22:50 GMT

On Dec 9, 2009, at 9:46 AM, Ludovico Boratto wrote:

> Hi everyone,
> I'm a PhD student, and I was wondering how is it possible to evaluate a search engine
performances with a dataset like the ones made available for the TREC tracks.
> 
> The problem I have is: once I submit a query and the search engine forms a list of ranked
results, how do I know which documents are relevant and the ones that are not?
> I know that with each track there are relevant judgments available, but those judgments
are about a small number of queries and documents.
> 

I think there is two questions in here, if I'm not mistaken.  First, TREC delivers a set of
qrels files that capture this information based on the way they do relevance pooling.  If
you search for trec_eval, you will find a tool that takes in your results and the TREC judgments
and outputs MAP, etc.

The second question I'm inferring is you want to know how does this apply to your search application.
 That is, how do you judge relevance for your app.   The answer to that is a bit harder. 
Essentially, you need to go through and create the queries and judgments.  Many people use
log analysis to achieve this.  You might find http://www.lucidimagination.com/Community/Hear-from-the-Experts/Articles/Debugging-Relevance-Issues-Search
helpful.


> Since these are my first steps in the IR world, I hope you don't mind helping me, please.
> 
> Thanks in advance for your help, I'm looking forward to hearing from you soon.
> Yours faithfully,
> Ludovico

--------------------------
Grant Ingersoll
http://www.lucidimagination.com/

Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids) using Solr/Lucene:
http://www.lucidimagination.com/search


Mime
View raw message