Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Message-Id: <1465547348.1320676.633590865.30E1F8F2@webmail.messagingengine.com>
From: Upayavira <uv@odoko.co.uk>
To: solr-user@lucene.apache.org
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: text/plain
Subject: Re: Scoring changes between 4.10 and 5.5
Date: Fri, 10 Jun 2016 09:29:08 +0100
In-Reply-To: <766815039.473907.1465519169749.JavaMail.yahoo@mail.yahoo.com>
References: <1465508216.485906.633216361.574C43BA@webmail.messagingengine.com>
 <766815039.473907.1465519169749.JavaMail.yahoo@mail.yahoo.com>
archived-at: Fri, 10 Jun 2016 08:29:19 -0000

Tracked it down to this ticket:

https://issues.apache.org/jira/browse/LUCENE-6590

which changed the implementation of normalize() in
org.apache.lucene.search.similarities.TFIDFSimilarity.

I've asked for comment on that ticket.

Upayavira

On Fri, 10 Jun 2016, at 01:39 AM, Ahmet Arslan wrote:
> Hi,
> 
> I wondered the same before and failed to decipher TFIDFSimilarity.
> Scoring looks like tf*idf*idf to me.
> 
> I appreciate someone who will shed some light on this.
> 
> Thanks,
> Ahmet
> 
> 
> 
> On Friday, June 10, 2016 12:37 AM, Upayavira <uv@odoko.co.uk> wrote:
> I've just done a very simple, single term query against a 4.10 system
> and a 5.5 system, each with much the same data.
> 
> The score for the 4.10 system was essentially made up of the field
> weight, which is:
>    score = tf * idf 
> 
> Whereas, in the 5.5 system, there is an additional "query weight", which
> is idf * query norm. If query norm is 1, then the final score is now:
>   score = query_weight * field_weight
>           = ( idf * 1 ) * (tf * idf)
>           = tf * idf^2
> 
> Can anyone explain why this new "query weight" element has appeared in
> our scores somewhere between 4.10 and 5.5?
> 
> Thanks!
> 
> Upayavira
> 
> 4.10 score ========================================================
>       "2937439": {
>         "match": true,
>         "value": 5.5993805,
>         "description": "weight(description:obama in 394012)
>         [DefaultSimilarity], result of:",
>         "details": [
>           {
>             "match": true,
>             "value": 5.5993805,
>             "description": "fieldWeight in 394012, product of:",
>             "details": [
>               {
>                 "match": true,
>                 "value": 1,
>                 "description": "tf(freq=1.0), with freq of:",
>                 "details": [
>                   {
>                     "match": true,
>                     "value": 1,
>                     "description": "termFreq=1.0"
>                   }
>                 ]
>               },
>               {
>                 "match": true,
>                 "value": 5.5993805,
>                 "description": "idf(docFreq=56010, maxDocs=5568765)"
>               },
>               {
>                 "match": true,
>                 "value": 1,
>                 "description": "fieldNorm(doc=394012)"
>               }
>             ]
>           }
>         ]
> 5.5 score ========================================================
>       "2502281":{
>         "match":true,
>         "value":28.51136,
>         "description":"weight(description:obama in 43472) [], result
>         of:",
>         "details":[{
>             "match":true,
>             "value":28.51136,
>             "description":"score(doc=43472,freq=1.0), product of:",
>             "details":[{
>                 "match":true,
>                 "value":5.339603,
>                 "description":"queryWeight, product of:",
>                 "details":[{
>                     "match":true,
>                     "value":5.339603,
>                     "description":"idf(docFreq=31905,
>                     maxDocs=2446459)"},
>                   {
>                     "match":true,
>                     "value":1.0,
>                     "description":"queryNorm"}]},
>               {
>                 "match":true,
>                 "value":5.339603,
>                 "description":"fieldWeight in 43472, product of:",
>                 "details":[{
>                     "match":true,
>                     "value":1.0,
>                     "description":"tf(freq=1.0), with freq of:",
>                     "details":[{
>                         "match":true,
>                         "value":1.0,
>                         "description":"termFreq=1.0"}]},
>                   {
>                     "match":true,
>                     "value":5.339603,
>                     "description":"idf(docFreq=31905,
>                     maxDocs=2446459)"},
>                   {
>                     "match":true,
>                     "value":1.0,
>                     "description":"fieldNorm(doc=43472)"}]}]}]},