lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Markus Jelsma <markus.jel...@openindex.io>
Subject Multiple languages, boosting and, stemming and KeywordRepeat
Date Wed, 09 May 2018 15:39:33 GMT
Hello,

First, apologies for the weird subject line.

We index many languages and search over all those languages at once, but boost the language
of the user's preference. To differentiate between stemmed tokens and unstemmed tokens we
use KeywordRepeat and RemoveDuplicates, this works very well.

However, we just stumbled over the following example, q=australia is not stemmed in English,
but its suffix is removed by the Romanian stemmer, causing the Romanian results to be returned
on top of English results, despite language boosting.

This is because the Romanian part of the query consists of the stemmed and unstemmed version
of the word, but the English part of the query is just one clause per field (title, content
etc). Thus the Romanian results score roughtly twice that of English results.

Now, this is of course really obvious, but the 'solution' is not. To work around the problem
i removed the RemoveDuplicates filter so i get two clauses for English as well, really ugly
but it works. What i don't understand is the debug output, it doesn't list two identical clauses,
instead, it doubled the boost on the field, so instead of:

    27.048403 = PayloadSpanQuery, product of:
      27.048403 = weight(title_en:australia in 15850) [SchemaSimilarity], result of:
        27.048403 = score(doc=15850,freq=4.0 = phraseFreq=4.0
), product of:
          7.4 = boost
          3.084852 = idf(docFreq=14539, docCount=317894)
          1.1848832 = tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * fieldLength
/ avgFieldLength)) from:
            4.0 = phraseFreq=4.0
            0.3 = parameter k1
            0.5 = parameter b
            15.08689 = avgFieldLength
            24.0 = fieldLength
      1.0 = AveragePayloadFunction.docScore()

I now get 

    54.096806 = PayloadSpanQuery, product of:
      54.096806 = weight(title_en:australia in 15850) [SchemaSimilarity], result of:
        54.096806 = score(doc=15850,freq=4.0 = phraseFreq=4.0
), product of:
          14.8 = boost
          3.084852 = idf(docFreq=14539, docCount=317894)
          1.1848832 = tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * fieldLength
/ avgFieldLength)) from:
            4.0 = phraseFreq=4.0
            0.3 = parameter k1
            0.5 = parameter b
            15.08689 = avgFieldLength
            24.0 = fieldLength
      1.0 = AveragePayloadFunction.docScore()

So instead of expecting two clauses in the debug, i get one but with a doubled boost.

The question is, is this supposed to be like this?

Also, are there any real solutions to this problem? Removing the RemoveDuplicats filter looks
really silly.

Many thanks!
Markus

Mime
View raw message