lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yuval Kesten <ykes...@yahoo-inc.com>
Subject RE: Custom lucene scoring - Dot product between field boost and query boost
Date Wed, 22 Feb 2012 17:28:42 GMT
Hi all,
Inspired by another thread here (Question about CustomScoreQuery) I am using this solution
which is working really well (with one drawback):
I discovered that some of my problems were due to the fact that my assumption was wrong:
I did have many fields/queries terms with the same field ID.
This ruined my approach because the query boost was aggregated and my calculations were wrong.

What I did was during indexing I added the field value to the field id (concatenated it by
'_') and as filed value used the desired score.

At search time I am using simple FieldScoreQuery (As-is, no modifications needed) with the
complex field ID.
Here I can still use the setBoost to set the score because now my filed are unique.

Logic wise this is perfect - dot product using Lucene.

Drawback - Lots of lots of different types of fields - effects the memory usage dramatically.

If anyone has better ideas - please share!

-----Original Message-----
From: Alan Woodward [mailto:alan.woodward@romseysoftware.co.uk] 
Sent: Wednesday, February 22, 2012 4:00 PM
To: java-user@lucene.apache.org
Subject: Re: Custom lucene scoring - Dot product between field boost and query boost

Hi Yuval,

You can just override Similarity, rather than DefaultSimilarity - that way you don't burn
any CPU cycles on TF/IDF calculations.

Alan

On 22 Feb 2012, at 07:17, Yuval Kesten wrote:

> Hi Em,
> 1. Regarding the performances - the similarity class (And my subtype as well) gets the
IDF and TF and SQUARED SUMS calculations as inputs - they just factor them differently. Even
though I ignore the values they are being computed.
> 2. I have written this code:
>    static {
>        Similarity.setDefault(new MySimilarity());
>    }
> Which means that I am setting the default similarity before doing the indexing and obviously
before the searching.
> Thanks!
> 
> -----Original Message-----
> From: Em [mailto:mailformailinglists@yahoo.de]
> Sent: Tuesday, February 21, 2012 6:07 PM
> To: java-user@lucene.apache.org
> Subject: Re: Custom lucene scoring - Dot product between field boost 
> and query boost
> 
> Hi Yuval,
> 
>> 1. Performances: I am calculating all the TF/IDF stuff and NORMS for 
>> nothing...
> You aren't calculating that much, since you declared all those values as constants. What
are you worried about?
> 
>> 2. The score I get from the TopScoreDocCollector is not the same as I
> get from the Explanation.
>> Here is part of my code:
> Could you provide us the code where you are setting the Similarity, please?
> 
> Kind regards,
> Em
> 
> Am 21.02.2012 16:18, schrieb Yuval Kesten:
>> Hi,
>> I want to use Lucene with the following scoring logic:
>> When I index my documents I want to set for each field a score/weight.
>> When I query my index I want to set for each query term a score/weight.
>> 
>> I will NEVER index or query with many instances of the same field - In each query
(document) there will be 0-1 instances with the same field name.
>> My fields/query term are not analyzed - they are already made out of one token.
>> 
>> I want the score to be simply the dot product between the fields of the query to
the fields of the document if they have the same value.
>> 
>> For example:
>> Query:
>> Field Name
>> 
>> Field Value
>> 
>> Field Score
>> 
>> 1
>> 
>> AA
>> 
>> 0.1
>> 
>> 7
>> 
>> BB
>> 
>> 0.2
>> 
>> 8
>> 
>> CC
>> 
>> 0.3
>> 
>> 
>> Document 1:
>> Field Name
>> 
>> Field Value
>> 
>> Field Score
>> 
>> 1
>> 
>> AA
>> 
>> 0.2
>> 
>> 2
>> 
>> DD
>> 
>> 0.8
>> 
>> 7
>> 
>> CC
>> 
>> 0.999
>> 
>> 10
>> 
>> FFF
>> 
>> 0.1
>> 
>> 
>> Document 2:
>> Field Name
>> 
>> Field Value
>> 
>> Field Score
>> 
>> 7
>> 
>> BB
>> 
>> 0.3
>> 
>> 8
>> 
>> CC
>> 
>> 0.5
>> 
>> 
>> The scores should be:
>> Score(q,d1) = FIELD_1_SCORE_Q * FILED_1_SCORE_D1 = 0.1 * 0.2  = 0.02
>> Score(q,d2) = FIELD_7_SCORE_Q * FILED_7_SCORE_D2 + FIELD_8_SCORE_Q *
>> FILED_8_SCORE_D2 = (0.2 * 0.3) + (0.3 * 0.5)
>> 
>> What would be the best way implement it? In terms of accuracy and performances (I
don't need TF and IDF calculations).
>> 
>> I currently implemented it by setting boosts to the fields and query terms.
>> Then I overwritten the DefaultSimilarity class:
>> 
>> public class MySimilarity extends DefaultSimilarity {
>> 
>>    @Override
>>    public float computeNorm(String field, FieldInvertState state) {
>>        return state.getBoost();
>>    }
>> 
>>    @Override
>>    public float queryNorm(float sumOfSquaredWeights) {
>>        return 1;
>>    }
>> 
>>    @Override
>>    public float tf(float freq) {
>>        return 1;
>>    }
>> 
>>    @Override
>>    public float idf(int docFreq, int numDocs) {
>>        return 1;
>>    }
>> 
>>    @Override
>>    public float coord(int overlap, int maxOverlap) {
>>        return 1;
>>    }
>> 
>> }
>> 
>> And based on http://lucene.apache.org/core/old_versioned_docs/versions/3_5_0/scoring.html
this should work.
>> Problems:
>> 1. Performances: I am calculating all the TF/IDF stuff and NORMS for nothing...
>> 2. The score I get from the TopScoreDocCollector is not the same as I get from the
Explanation.
>> Here is part of my code:
>> 
>> indexSearcher = new IndexSearcher(IndexReader.open(directory, true)); 
>> TopScoreDocCollector collector = TopScoreDocCollector.create(iTopN,
>> true); indexSearcher.search(query, collector); ScoreDoc[] hits = 
>> collector.topDocs().scoreDocs; for (int i = 0; i < hits.length; ++i) 
>> { int docId = hits[i].doc; Document d = indexSearcher.doc(docId); 
>> double score = hits[i].score; String id = d.get(FIELD_ID); 
>> Explanation explanation = indexSearcher.explain(query, docId); }
>> 
>> Thanks!
>> 
>> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message