lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anshum <ansh...@gmail.com>
Subject Re: boosting relevance of certain documents
Date Sat, 26 Apr 2008 04:32:56 GMT
Hi Daniel,

Just a suggestion, how bout storing an extra field while indexing that has
the "length" of the document. You could just divide the score of the
document (change the lucene code) with the length of the document (or
something on the same lines) while calculating the score. In this manner,
among 2 docs, the smaller doc with the same score would get preference.
Do you think that would somehow solve your problem?
Though again, it involves changing the algo but this would be useful in case
you have documents that keep on getting updated and you can not afford to
hard-code the doc preference.

--
Anshum Gupta


On Sat, Apr 26, 2008 at 4:12 AM, Grant Ingersoll <gsingers@apache.org>
wrote:

> It really depends.  Hand tuning scoring algs for a specific query is very
> prone to local maxima problems.  In other words, you fix one query and break
> 50 others.  Sometimes, a good old "configurable" hard code is the way to go.
>  If you want a specific doc to be #1, make it number one.  You will pull
> your hair out otherwise.  In Solr, this is handled via the Query Elevation
> Component, but isn't all that difficult to implement.
>
> Likewise, if you have a priori knowledge that a particular document is
> more important, then give it a relatively large boost during indexing, being
> aware that Lucene does not offer much granularity in terms of boosts.  In
> other words, boost it something like 5 or 10 times, instead of 1.1 vs. 1.2.
>
> On the other hand, if you are truly seeing broad problems, then you need
> to build up a set of queries and judgments (ala TREC) or the
> contrib/benchmark Quality packages.  You might also look at Lucene's
> Similarity class.  Lucene's length normalization is often less than optimal
> for certain types of documents (see the IBM Haifa's assessment for the
> "Million Query" track of TREC on the Lucene Wiki).
>
> Cheers,
> Grant
>
>
> On Apr 25, 2008, at 3:50 PM, Daniel Freudenberger wrote:
>
>  Thanks for your response. I already knew that the relevance is based on
> > the
> > term frequency but in some cases it's just not what the user expects.
> > As I already mentioned, "fifa 2003 fifa 03" vs. "fifa 08" is such a case
> > -
> > searching for "fifa" would return the "fifa 2003 fifa 03" document first
> > but
> > the "fifa 08" document is more important (from the user's point of
> > view).
> >
> > Any suggestions?
> >
> > Best regards,
> > Daniel
> > -----Original Message-----
> > From: Jonathan Ariel [mailto:ionathan@gmail.com]
> > Sent: Friday, April 25, 2008 8:11 PM
> > To: java-user@lucene.apache.org
> > Subject: Re: boosting relevance of certain documents
> >
> > Ok. So I'm not an expert of the scoring algorithm, but based on tf*idf
> > you
> > can tell that the returned document is more relevant because it has more
> > term frequency.
> >
> > Using the explain you can see the following:
> >
> > Doc 1
> > 0.643841 = (MATCH) fieldWeight(searchable:fifa in 0), product of:
> >  1.0 = tf(termFreq(searchable:fifa)=1)
> >  1.287682 = idf(docFreq=2)
> >  0.5 = fieldNorm(field=searchable, doc=0)
> >
> > Doc2
> > 0.68289655 = (MATCH) fieldWeight(searchable:fifa in 1), product of:
> >  1.4142135 = tf(termFreq(searchable:fifa)=2)
> >  1.287682 = idf(docFreq=2)
> >  0.375 = fieldNorm(field=searchable, doc=1)
> >
> > On Fri, Apr 25, 2008 at 2:30 PM, Daniel Freudenberger <
> > d.freudenberger@trade-a-game.de> wrote:
> >
> >  I'm using the StandardAnalyzer - hope this answers your question (I'm
> > > quite
> > > new to the lucene thing)
> > >
> > > -----Original Message-----
> > > From: Jonathan Ariel [mailto:ionathan@gmail.com]
> > > Sent: Friday, April 25, 2008 6:59 PM
> > > To: java-user@lucene.apache.org
> > > Subject: Re: boosting relevance of certain documents
> > >
> > > How are you analyzing the searchable field?
> > >
> > > On Fri, Apr 25, 2008 at 12:49 PM, Daniel Freudenberger <
> > > d.freudenberger@trade-a-game.de> wrote:
> > >
> > >  Hello,
> > > >
> > > >
> > > >
> > > > I'm using lucene within a new project and I'm not sure about how to
> > > >
> > > solve
> > >
> > > > the following problem: My index consists of the two attributes "id"
> > > > and
> > > > "searchable". "id" is the id of a product and "searchable" is a
> > > > combination
> > > > of the product name and its category name.
> > > >
> > > >
> > > >
> > > > example:
> > > >
> > > > id     searchable
> > > >
> > > > 1     fifa 08 - playstation 3
> > > >
> > > > 2     fifa 2003 fifa 03 - playstation 3
> > > >
> > > > 3     playstation 60gb hdd - playstation 3
> > > >
> > > > 4     playstation i like you - playstation 3
> > > >
> > > >
> > > >
> > > > When searching for "fifa", lucene returns the product with id 2 at
> > > >
> > > first,
> > >
> > > > whereas id 1 ("fifa 08") would be the much more relevant result
> > > > (from
> > > >
> > > the
> > >
> > > > user side of view). the same problem arises when searching for
> > > > "playstation"
> > > > - the customer expects products having "playstation" in their names
> > > > at
> > > > first, ideally the console itself. in reality however, he gets all
> > > > possible
> > > > products which are in the "playstation" category as well.
> > > >
> > > >
> > > >
> > > > my idea was to introduce another attribute relevance, which may
> > > > increase
> > > > the
> > > > relevance of an entry. the actual relevance shouldn't be suppressed
> > > > completely though, but should only be taken into account with
> > > > products
> > > > that
> > > > are similarly relevant for a specific search term.
> > > >
> > > >
> > > >
> > > > Does anybody have an idea on how to solve this problem?
> > > >
> > > >
> > > >
> > > > Thank you in advance,
> > > >
> > > > Daniel
> > > >
> > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
> --------------------------
> Grant Ingersoll
>
> Lucene Helpful Hints:
> http://wiki.apache.org/lucene-java/BasicsOfPerformance
> http://wiki.apache.org/lucene-java/LuceneFAQ
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>


-- 
--
The facts expressed here belong to everybody, the opinions to me.
The distinction is yours to draw............

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message