lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mikhail Khludnev <>
Subject Re: Pagination bug? when sorting by a field (not unique field)
Date Wed, 29 Mar 2017 20:14:22 GMT
Great explanation, Alessandro!

Let me briefly explain my experience. I have a tiny test with 2 shards and
2 replicas, index about a hundred of docs. And then when I fully paginate
search results with score ranking, I've got duplicates across pages. And
the reason is deletes, which occur probably due to update/failover. Every
paging request lands to the different replica. There are a few workarounds:
lands consequent requests to the same replicas; also <optimize> fixes
duplicates; but tie-breaking is the best way for sure.

On Wed, Mar 29, 2017 at 7:10 PM, alessandro.benedetti <>

> The reason Mikhail mentioned that, is probably related to :
> *The way how number of document calculated is changed (LUCENE-6711)*
> /The number of documents (docCount) is used to calculate term specificity
> (idf) and average document length (avdl). Prior to LUCENE-6711,
> collectionStats.maxDoc() was used for the statistics. Now,
> collectionStats.docCount() is used whenever possible, if not maxDocs() is
> used.
> Assume that a collection contains 100 documents, and 50 of them have
> "keywords" field. In this example, maxDocs is 100 while docCount is 50 for
> the "keywords" field. The total number of tokens for "keywords" field is
> divided by docCount to obtain avdl. Therefore, docCount which is the total
> number of documents that have at least one term for the field, is a more
> precise metric for optional fields.
> DefaultSimilarity does not leverage avdl, so this change would have
> relatively minor change in the result list. Because relative idf values of
> terms will remain same. However, when combined with other factors such as
> term frequency, relative ranking of documents could change. Some Similarity
> implementations (such as the ones instantiated with NormalizationH2 and
> BM25) take account into avdl and would have notable change in ranked list.
> Especially if you have a collection of documents with varying lengths.
> Because NormalizationH2 tends to punish documents longer than avdl./
> This means that if you are load balancing, the page 2 query could go to
> another replica, where the doc is scored differently, ending up on a
> different position ( and maybe appearing again as a final effect).
> This scenario is referred to scored ranking, so it will not affect sorting
> (
> and I believe in your initial mail you were referring not to sorting)
> Cheers
> Pablo wrote
> > Mikhall,
> >
> > effectively maxDocs are different and also deletedDocs, but numDocs are
> > ok.
> >
> > I don't really get it, but can that be the problem?
> -----
> ---------------
> Alessandro Benedetti
> Search Consultant, R&D Software Engineer, Director
> Sease Ltd. -
> --
> View this message in context: http://lucene.472066.n3.
> tp4327408p4327461.html
> Sent from the Solr - User mailing list archive at

Sincerely yours
Mikhail Khludnev

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message