lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Muir <rcm...@gmail.com>
Subject Re: Average Precision - TREC-3
Date Thu, 28 Jan 2010 16:00:33 GMT
in addition to what Grant said, even if your documents are similar, what
about queries?

For example, if only a few trec queries contain proper names, acronyms,
abbreviations, or whatever, but your users frequently input things like
this, it won't be representative.

i will disagree with him on a few things though, I would rather have less
queries (25 or so), but more judgements, definitely a lot more than 10.
Maybe your users only care about the top-10 results but its crucial to judge
some lower-ranking docs too, especially if you have recall problems...

On Thu, Jan 28, 2010 at 9:34 AM, Grant Ingersoll <gsingers@apache.org>wrote:

>
> On Jan 27, 2010, at 1:36 PM, Ivan Provalov wrote:
>
> > Robert, Grant:
> >
> > Thank you for your replies.
> >
> > Our goal is to fine-tune our existing system to perform better on
> relevance.
>
> What kind of documents do you have?  Are they very similar to the TREC docs
> (i.e. news articles)?  There can be a fairly wide difference in performance
> between real docs and TREC docs, especially given real queries.  Doing well
> at TREC does not necessarily equate to doing well in your own system.  You
> might be better off just doing something like taking the top 50 queries from
> your logs plus some random ones from the tail and judging the top 10.   See
> http://www.lucidimagination.com/Community/Hear-from-the-Experts/Articles/Debugging-Relevance-Issues-Search
>
>
> --------------------------
> Grant Ingersoll
> http://www.lucidimagination.com/
>
> Search the Lucene ecosystem using Solr/Lucene:
> http://www.lucidimagination.com/search
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>


-- 
Robert Muir
rcmuir@gmail.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message