lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chuck Williams" <ch...@manawiz.com>
Subject RE: Similarity coord,lengthNorm
Date Mon, 07 Feb 2005 16:09:23 GMT
Hi Michael,

I'd suggest first using the explain() mechanism to figure out what's
going on.  Besides lengthNorm(), another factor that is likely skewing
your results in my experience is idf(), which Lucene typically makes
very large by squaring the intrinsic value.  I've found it helpful to
flatten lengthNorm(), tf() and idf() relative to what is used in
DefaultSimilarity.  There is a comparative evaluation of Similarity's
going on now.  You might consider looking at these:

Bug 32674 has a WikipediaSimilarity posted that you might want to try.
You might want to flatten lengthNorm() even further (e.g. all the way to
1.0), but I'd suggest trying it as is first.  If you try it, please post
your assessment.  Here's the link:
http://issues.apache.org/bugzilla/show_bug.cgi?id=32674

You also might find it interesting to read the thread entitled "RE:
Scoring benchmark evaluation.  Was RE: How to proceed with Bug 31841 -
MultiSearcher problems with Similarity.docFreq() ?" on lucene-dev, as
this contains a discussion of many of the issues.

Good luck,

Chuck

  > -----Original Message-----
  > From: Erik Hatcher [mailto:erik@ehatchersolutions.com]
  > Sent: Monday, February 07, 2005 6:51 AM
  > To: Lucene Users List
  > Subject: Re: Similarity coord,lengthNorm
  > 
  > 
  > On Feb 7, 2005, at 8:53 AM, Michael Celona wrote:
  > > Would fixing the lengthNorm to 1 fix this problem?
  > 
  > Yes, it would eliminate the length of a field as a factor.
  > 
  > Your best bet is to set up a test harness where you can try out
various
  > tweaks to Similarity, but setting the length normalization factor to
  > 1.0 may be all you need to do, as the coord() takes care of the
other
  > factor you're after.
  > 
  > 	Erik
  > 
  > >
  > > Michael
  > >
  > > -----Original Message-----
  > > From: Michael Celona [mailto:mcelona@criticalmention.com]
  > > Sent: Monday, February 07, 2005 8:48 AM
  > > To: Lucene Users List
  > > Subject: Similarity coord,lengthNorm
  > >
  > > I have varying length text fields which I am searching on.  I
would
  > > like
  > > relevancy to be dictated predominantly by the number of terms in
my
  > > query
  > > that match.  Right now I am seeing a high relevancy for a single
word
  > > matching in a small document even though all the terms in my query
  > > don't
  > > match.  Does, anyone have an example of a custom Similarity sub
class
  > > which
  > > overrides the coord and lengthNorm methods.
  > >
  > >
  > >
  > > Thanks..
  > >
  > > Michael
  > >
  > >
  > >
  > >
  > >
---------------------------------------------------------------------
  > > To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
  > > For additional commands, e-mail:
lucene-user-help@jakarta.apache.org
  > 
  > 
  >
---------------------------------------------------------------------
  > To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
  > For additional commands, e-mail: lucene-user-help@jakarta.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message