lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Question about threading in search
Date Fri, 17 Aug 2018 14:58:49 GMT
Please don't optimize to 1 segment unless you can afford to do it
quite regularly, see:
https://lucidworks.com/2017/10/13/segment-merging-deleted-documents-optimize-may-bad/

(NOTE: this is changing as of 7.5, see:
https://lucidworks.com/2018/06/20/solr-and-optimizing-your-index-take-ii/).

bq. It seems that search is sometimes faster with multiple segments.

In addition to what Toke said, this may just be an autowarming
problem. Measurements mean little/nothing unless they're performed on
a warmed-up index since there's quite a bit of reading from disk into
the heap and OS memory space that's required. You may just be seeing
that.

Best,
Erick


On Fri, Aug 17, 2018 at 2:26 AM, Toke Eskildsen <toes@kb.dk> wrote:
> On Sat, 2017-09-02 at 18:33 -0700, Peilin Yang wrote:
>> we're comparing two different indexes on the same collection - one
>> with lots of different segments (default settings), and one with a
>> force merged into one segment. It seems that search is sometimes
>> faster with multiple segments.
>
> If you are using Lucene 7+ and if some of the fields you are requesting
> as part of your search result are stored as DocValues, you might have
> encountered a performance regression with the streaming API:
> https://issues.apache.org/jira/browse/LUCENE-8374
>
> One peculiar effect of this issue is that fewer larger segments gets
> slower DocValues retrieval, compared to more smaller segments. So a
> force merge to 1 segment can result in worse performance.
>
> - Toke Eskildsen, the Royal Danish Library, Denmark
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message