lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis_gospodne...@yahoo.com>
Subject Re: boosting relevance of certain documents
Date Sat, 26 Apr 2008 05:02:33 GMT
If this is really about adjusting score based on field length (didn't follow the thread closely),
then this sounds like a job for a custom Similarity with a custom implementation of lengthNorm
method.

Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch

----- Original Message ----
> From: Anshum <anshumg@gmail.com>
> To: java-user@lucene.apache.org
> Sent: Saturday, April 26, 2008 12:32:56 AM
> Subject: Re: boosting relevance of certain documents
> 
> Hi Daniel,
> 
> Just a suggestion, how bout storing an extra field while indexing that has
> the "length" of the document. You could just divide the score of the
> document (change the lucene code) with the length of the document (or
> something on the same lines) while calculating the score. In this manner,
> among 2 docs, the smaller doc with the same score would get preference.
> Do you think that would somehow solve your problem?
> Though again, it involves changing the algo but this would be useful in case
> you have documents that keep on getting updated and you can not afford to
> hard-code the doc preference.
> 
> --
> Anshum Gupta
> 
> 
> On Sat, Apr 26, 2008 at 4:12 AM, Grant Ingersoll 
> wrote:
> 
> > It really depends.  Hand tuning scoring algs for a specific query is very
> > prone to local maxima problems.  In other words, you fix one query and break
> > 50 others.  Sometimes, a good old "configurable" hard code is the way to go.
> >  If you want a specific doc to be #1, make it number one.  You will pull
> > your hair out otherwise.  In Solr, this is handled via the Query Elevation
> > Component, but isn't all that difficult to implement.
> >
> > Likewise, if you have a priori knowledge that a particular document is
> > more important, then give it a relatively large boost during indexing, being
> > aware that Lucene does not offer much granularity in terms of boosts.  In
> > other words, boost it something like 5 or 10 times, instead of 1.1 vs. 1.2.
> >
> > On the other hand, if you are truly seeing broad problems, then you need
> > to build up a set of queries and judgments (ala TREC) or the
> > contrib/benchmark Quality packages.  You might also look at Lucene's
> > Similarity class.  Lucene's length normalization is often less than optimal
> > for certain types of documents (see the IBM Haifa's assessment for the
> > "Million Query" track of TREC on the Lucene Wiki).
> >
> > Cheers,
> > Grant
> >
> >
> > On Apr 25, 2008, at 3:50 PM, Daniel Freudenberger wrote:
> >
> >  Thanks for your response. I already knew that the relevance is based on
> > > the
> > > term frequency but in some cases it's just not what the user expects.
> > > As I already mentioned, "fifa 2003 fifa 03" vs. "fifa 08" is such a case
> > > -
> > > searching for "fifa" would return the "fifa 2003 fifa 03" document first
> > > but
> > > the "fifa 08" document is more important (from the user's point of
> > > view).
> > >
> > > Any suggestions?
> > >
> > > Best regards,
> > > Daniel
> > > -----Original Message-----
> > > From: Jonathan Ariel [mailto:ionathan@gmail.com]
> > > Sent: Friday, April 25, 2008 8:11 PM
> > > To: java-user@lucene.apache.org
> > > Subject: Re: boosting relevance of certain documents
> > >
> > > Ok. So I'm not an expert of the scoring algorithm, but based on tf*idf
> > > you
> > > can tell that the returned document is more relevant because it has more
> > > term frequency.
> > >
> > > Using the explain you can see the following:
> > >
> > > Doc 1
> > > 0.643841 = (MATCH) fieldWeight(searchable:fifa in 0), product of:
> > >  1.0 = tf(termFreq(searchable:fifa)=1)
> > >  1.287682 = idf(docFreq=2)
> > >  0.5 = fieldNorm(field=searchable, doc=0)
> > >
> > > Doc2
> > > 0.68289655 = (MATCH) fieldWeight(searchable:fifa in 1), product of:
> > >  1.4142135 = tf(termFreq(searchable:fifa)=2)
> > >  1.287682 = idf(docFreq=2)
> > >  0.375 = fieldNorm(field=searchable, doc=1)
> > >
> > > On Fri, Apr 25, 2008 at 2:30 PM, Daniel Freudenberger <
> > > d.freudenberger@trade-a-game.de> wrote:
> > >
> > >  I'm using the StandardAnalyzer - hope this answers your question (I'm
> > > > quite
> > > > new to the lucene thing)
> > > >
> > > > -----Original Message-----
> > > > From: Jonathan Ariel [mailto:ionathan@gmail.com]
> > > > Sent: Friday, April 25, 2008 6:59 PM
> > > > To: java-user@lucene.apache.org
> > > > Subject: Re: boosting relevance of certain documents
> > > >
> > > > How are you analyzing the searchable field?
> > > >
> > > > On Fri, Apr 25, 2008 at 12:49 PM, Daniel Freudenberger <
> > > > d.freudenberger@trade-a-game.de> wrote:
> > > >
> > > >  Hello,
> > > > >
> > > > >
> > > > >
> > > > > I'm using lucene within a new project and I'm not sure about how
to
> > > > >
> > > > solve
> > > >
> > > > > the following problem: My index consists of the two attributes "id"
> > > > > and
> > > > > "searchable". "id" is the id of a product and "searchable" is a
> > > > > combination
> > > > > of the product name and its category name.
> > > > >
> > > > >
> > > > >
> > > > > example:
> > > > >
> > > > > id     searchable
> > > > >
> > > > > 1     fifa 08 - playstation 3
> > > > >
> > > > > 2     fifa 2003 fifa 03 - playstation 3
> > > > >
> > > > > 3     playstation 60gb hdd - playstation 3
> > > > >
> > > > > 4     playstation i like you - playstation 3
> > > > >
> > > > >
> > > > >
> > > > > When searching for "fifa", lucene returns the product with id 2 at
> > > > >
> > > > first,
> > > >
> > > > > whereas id 1 ("fifa 08") would be the much more relevant result
> > > > > (from
> > > > >
> > > > the
> > > >
> > > > > user side of view). the same problem arises when searching for
> > > > > "playstation"
> > > > > - the customer expects products having "playstation" in their names
> > > > > at
> > > > > first, ideally the console itself. in reality however, he gets all
> > > > > possible
> > > > > products which are in the "playstation" category as well.
> > > > >
> > > > >
> > > > >
> > > > > my idea was to introduce another attribute relevance, which may
> > > > > increase
> > > > > the
> > > > > relevance of an entry. the actual relevance shouldn't be suppressed
> > > > > completely though, but should only be taken into account with
> > > > > products
> > > > > that
> > > > > are similarly relevant for a specific search term.
> > > > >
> > > > >
> > > > >
> > > > > Does anybody have an idea on how to solve this problem?
> > > > >
> > > > >
> > > > >
> > > > > Thank you in advance,
> > > > >
> > > > > Daniel
> > > > >
> > > > >
> > > > >
> >



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message