lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chuck Williams" <>
Subject RE: Scoring benchmark evaluation. Was RE: How to proceed with Bug 31841 - MultiSearcher problems with Similarity.docFreq() ?
Date Fri, 28 Jan 2005 21:42:06 GMT

I just posted WikipediaSimilarity to Bug 32674.  I've also reviewed and
tested the port to Java 1.4 -- it's fine (although all the casts remind
me why I like 1.5 so much).  Thanks to Miles Barr for this port!

You don't want any of the test classes.  You just need these 4 classes:


WikipediaSimilarity can be placed in whatever package makes sense within
your app.  It assumes your fields are named "title" and "body" (only
"body" is explicitly referenced).  If the names are different, you can
change the single reference to "body" in lengthNorm().  I'm assuming
title is just the title, and body is just the article without its title
(i.e., the two fields are disjoint).  If the semantics of the fields are
different, please let me know as this would imply a need to change
things more fundamentally.

The Wikipedia attachment suggests the same initial values to try for
DEFAULT_BOOSTS as in my previous message.

I believe in tuning per application and these classes will almost
certainly require tuning for the Wikipedia as my collection is very

Please let me know when I can take a look at it.  It would be most
efficient if there is some way I could directly tune the parameters
(DEFAULT_BOOSTS and the formulas in WikipediaSimilarity).  Again, the
explain mechanism will be most helpful in doing this.



  > -----Original Message-----
  > From: Chuck Williams []
  > Sent: Friday, January 28, 2005 8:53 AM
  > To: Lucene Developers List
  > Subject: RE: Scoring benchmark evaluation. Was RE: How to proceed
  > Bug 31841 - MultiSearcher problems with Similarity.docFreq() ?
  > David Spencer wrote:
  >   > I'm on JDK 1.4.2_06 and Tomcat 4+. Had issues w/ the Tomcat
  > 1.5
  >   >   combo so I rolled back.
  > There have been issues with Tomcat 5.5, although supposedly the
  > version has them resolved.  I'm using Tomcat 5.0.28 with JDK
  > which has been solid -- no problems at all.  But your combo should
  > fine; I just want to verify Miles Barr's changes to remove the 1.5
  > dependencies from my classes.
  >   > The baseline will presumably use the default Lucene Similarity
  > Query
  >   > Parser.
  > I think the baseline should use Lucene's MultiFieldQueryParser to
  > the query to search both title and body fields, as this is
  > the current "out-of-the-box" solution.  Similarly, it should use
  > 1.4.3, the current official release; is this what you are using?
  > may be a desire to use the CVS HEAD instead, which I have never run
  > with.
  >   > For "your case" you'll need to tell me if I need to call
  >   > DistributingMultiFieldQueryParser or whatnot.
  > Yes, you need something like this:
  >   private static final String[] DEFAULT_FIELDS = {"title", "body"};
  >   private static final float[] DEFAULT_BOOSTS = {3.0f, 1.0f};
  >   DistributingMultiFieldQueryParser.parse(
  >     queryString, DEFAULT_FIELDS, DEFAULT_BOOSTS, new
  > StandardAnalyzer());
  > DEFAULT_FIELDS should contain whatever list of fields you are using
  > (that should be searched for simple query terms containing no
  > field specs).  DEFAULT_BOOSTS must be in 1:1 correspondence with
  >   > Also, dumb I need to build an index for every impl
  >   > Similarity? Thus there will be "n" indexes and wikipedia-sim.jsp
  > will
  >   > search in each one with the corresponding Similarity?
  > Yes, you will need a separate index for each Similarity, as some
  > computed from the similarity are stored in the index.
  > I'll send you a Similarity and an initial value for DEFAULT_BOOSTS
  > today or tomorrow.
  > Can you put up an explain mechanism to support tuning?  I'll want to
  > tune the DEFAULT_BOOSTS and various Similarity factors based on the
  > collection.
  > Thanks,
  > Chuck
  > To unsubscribe, e-mail:
  > For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message