lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karl Koch" <TheRan...@gmx.net>
Subject Re: Question about scoring normalisation
Date Sun, 06 Nov 2005 19:43:17 GMT
Hello Ira,

I am not sure if I know exactly what pivoted normalisation is. I can tell
you what I do, in the meantime I will have a look to your paper and I hope
that we can discuss this issue further.

I work in personalised searching where a second model - a user model - 
expresses extra relevance of a document; however from a different
prespective. I also value the use of TF/IDF to represent the content sided
view on content relevance. Therefore, I would like to combine both scores
(the one from Lucene and the one from my model) into a combined score.
Furthermore, I need to normalise this score so it cannot get higher than
1.0. At the moment, I have a visulisation scheme that works on this premise.

In Lucene, scores seem to be normalised only if they exceed the 1.0 maximum
border. If they do not exceed this max value, they are left where they are. 

Based on that I have now solved my problem to combine my two scores (Lucene
and mine) and if they exceed, I normalise the scores like Lucene does. I
think this is the most accurate think I could do in this case where I do not
violoate the overall meaning of scoring for the user. 

I realise that if I always normalise (to a 0.0 to 1.0 range) I will
introduce a dangerous feature that basically boosts a bunch of low scored
documents (e.g. all between 0.1 and 0.2) unnecessarily high (in this case
right up to between 0.5 and 1.0).

What would you say about that? Does is make sense?

Kind Regards,
Karl


> --- Ursprüngliche Nachricht ---
> Von: Ira Goldstein <iragoldstein@usa.net>
> An: "Karl Koch" <TheRanger@gmx.net>
> Betreff: Re: Question about scoring normalisation
> Datum: Sun, 06 Nov 2005 08:08:59 -0500
> 
> Karl --
>   Hi.  I've been thinking about adding a pivoted normalization to Lucene
> (see
> attached paper).  I've just started to look at the current code to see how
> the
> tf-idf (sum of squares?) has been implemented.  I wanted to do that before
> begining any coding?  
>   Is pivoted normalization the sort of thing you were asking about?
>   Take care
> --Ira
> 
> ------ Original Message ------
> Received: Sat, 05 Nov 2005 03:26:12 PM EST
> From: "Karl Koch" <TheRanger@gmx.net>
> To: "User " <java-user@lucene.apache.org>
> Subject: Question about scoring normalisation
> 
> > Hello all,
> > 
> > I am wondering how many of you actually work with own scoring mechanism
> > (overwriting Lucenes standard scoring) and how many of you do work on
> how
> to
> > normalise this score. 
> > 
> > I would like to add a second score on top of Lucenes TF/IDF score. The
> > resulting score is most likely higher then 1.0. However, the score
> should
> be
> > between 0.0 and 1.0. What is the best way to do that? If Lucene is
> > normalising its score (if no boosting is applied) to a maximium of 1.0,
> how
> > is this done (in Lucene 1.2 and/or beyond) ?
> > 
> > Regards,
> > Karl
> > 
> > 
> > -- 
> > Highspeed-Freiheit. Bei GMX supergünstig, z.B. GMX DSL_Cityflat,
> > DSL-Flatrate für nur 4,99 Euro/Monat*  http://www.gmx.net/de/go/dsl
> > 
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> > 
> > 
> 
> 
> 

-- 
Highspeed-Freiheit. Bei GMX supergünstig, z.B. GMX DSL_Cityflat,
DSL-Flatrate für nur 4,99 Euro/Monat*  http://www.gmx.net/de/go/dsl

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message