lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chuck Williams" <ch...@manawiz.com>
Subject RE: URL to compare 2 Similarity's ready-- Re: Scoring benchmark evaluation. Was RE: How to proceed with Bug 31841 - MultiSearcher problems with Similarity.docFreq() ?
Date Mon, 31 Jan 2005 20:47:13 GMT
David, thanks for all your work on this!

I'm curious to see what others think.  To evaluate this properly would
require a reasonably systematic analysis of a significant number of
queries.  On the 4 test queries pre-selected on the page, I think the
new approach clearly performs better on "chess champion" and "russian
politics", and arguably performs better on "information retrieval search
engine".  "conspiracy theory" is basically a wash, and fairly
uninteresting since the terms appear widely in both titles and content.

"information retrieval search engine" is an interesting example.  First,
we should forget that we as people understand a close relationship
between "information retrieval" and "search engine", since neither
approach is using an ontology, nor even any correlation statistics
(e.g., LSI).  Then, look at the score for the SMART system result in the
current code.  The article does not mention "search" or "engine" and yet
it gets a very high score because "information retrieval" occurs in both
title and content.  This is a great example of the current approach's
failure to assess term diversity (i.e., coverage of distinct terms in
the query).

Chuck

  > -----Original Message-----
  > From: David Spencer [mailto:dave-lucene-dev@tropo.com]
  > Sent: Monday, January 31, 2005 11:35 AM
  > To: Lucene Developers List
  > Subject: URL to compare 2 Similarity's ready-- Re: Scoring benchmark
  > evaluation. Was RE: How to proceed with Bug 31841 - MultiSearcher
  > problems with Similarity.docFreq() ?
  > 
  > 
  > I worked w/ Chuck to get up a test page that shows search results
with 2
  > versions of Similarity side by side.
  > 
  > URL here:
  > 
  > 	http://www.searchmorph.com/kat/wikipedia-similarity.jsp
  > 
  > Weblog entry here w/ some more details:
  > 
  > 	http://www.searchmorph.com/weblog/index.php?id=46
  > 
  > 
  > But briefly the page uses 2 indexes of the wikipedia.
  > First index is all default Lucene code, and ditto for the query
parser.
  > 
  > The second index uses Chuck's suggestion for another similarity
  > implementation, and the search results use this same similarity +
the
  > query parser (DistributingMultiFieldQueryParser) he has proposed.
  > 
  > The page lets you tune parameters to his Similarity impl so you can
see
  > the effect of different weights.
  > 
  > One test that seems to show how the new code performs better is the
  > search for "russian politics" where the results on the right seem
more
  > relevant:
  > 
  > http://www.searchmorph.com/kat/wikipedia-
  > similarity.jsp?s=russian+politics
  > 
  > 
  > 
  > 
  > 
  > 
  > 
  > Chuck Williams wrote:
  > 
  > > Dave, are you using MultiFieldQueryParser and DefaultSimilarity
for
  > the
  > > vanilla implementation?
  > >
  > > It's important to know what we are comparing...
  > >
  > > Chuck
  > >
  > >   > -----Original Message-----
  > >   > From: David Spencer [mailto:dave-lucene-dev@tropo.com]
  > >   > Sent: Friday, January 28, 2005 3:38 PM
  > >   > To: Lucene Developers List
  > >   > Subject: Re: Scoring benchmark evaluation. Was RE: How to
proceed
  > > with
  > >   > Bug 31841 - MultiSearcher problems with Similarity.docFreq() ?
  > >   >
  > >   > Daniel Naber wrote:
  > >   >
  > >   > > On Friday 28 January 2005 22:45, Chuck Williams wrote:
  > >   > >
  > >   > >
  > >   > >>The fact that is requires all terms in all
  > >   > >>fields is part of the problem.  Once that is addressed,
another
  > >   > problem
  > >   > >>is that Lucene does not provide a good mechanis
  > >   > >
  > >   > >
  > >   > > That's fixed in CVS, so maybe the CVS version should be used
for
  > > the
  > >   > > evaluation. I think it should be robust.
  > >   >
  > >   > Hmmm, is it safe to assume I can build the index w/ lucene-
  > 1.4.3.jar
  > > but
  > >   >    deploy the webapp for searching w/ lucene-1.5-rc1-dev.jar?
  > >   >
  > >   > And is the current code supposed to build with so many
deprecated
  > >   > warnings?
  > >   >
  > >   > - Dave
  > >   >
  > >   > >
  > >   > > Regards
  > >   > >  Daniel
  > >   > >
  > >   >
  > >   >
  > >   >
  > >
---------------------------------------------------------------------
  > >   > To unsubscribe, e-mail:
lucene-dev-unsubscribe@jakarta.apache.org
  > >   > For additional commands, e-mail: lucene-dev-
  > help@jakarta.apache.org
  > >
  > >
  > >
---------------------------------------------------------------------
  > > To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
  > > For additional commands, e-mail:
lucene-dev-help@jakarta.apache.org
  > >
  > 
  > 
  >
---------------------------------------------------------------------
  > To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
  > For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Mime
View raw message