lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arkadi Colson <ark...@smartbit.be>
Subject Re: Merge policy
Date Fri, 28 Oct 2016 07:20:10 GMT
The index size of 1 shard is about 125GB and we are running 11 shards 
with replication factor 2 so it's a lot of data. The deletions 
percentage at the bottom of the segment page is around 25%. So it's 
quite some space which we could recover. That's why I was looking for an 
optimize.

Do you have any idea why the merge policy does not merge away the 
deletions? Should I tweak some parameters somehow? It's a default 
installation using the default settings and parameters. If you need more 
info, just let me know...

Thx!


On 27-10-16 17:40, Erick Erickson wrote:
> Why do you think you need to get rid of the deleted data? During normal
> indexing, these will be "merged away". Optimizing has some downsides
> for continually changing indexes, in particular since the default
> tieredmergepolicy tries to merge "like size" segments, deletions will
> accumulate in your one large segment and the percentage of
> deleted documents may get even higher.
>
> Unless there's some measurable performance gain that the users
> will notice, I'd just leave this alone.
>
> The exception here is if you have, say, an index that changes rarely
> in which case optimizing then makes more sense.
>
> Best,
> Erick
>
> On Thu, Oct 27, 2016 at 6:56 AM, Arkadi Colson <arkadi@smartbit.be 
> <mailto:arkadi@smartbit.be>> wrote:
>
>     Thanks for the answer!
>     Do you know if there is a way to trigger an optimize for only 1
>     shard and not the whole collection at once?
>
>
>     On 27-10-16 15:30, Pushkar Raste wrote:
>>
>>     Try commit with expungeDeletes="true"
>>
>>     I am not sure if it will merge old segments that have deleted
>>     documents.
>>
>>     In the worst case you can 'optimize' your index which should take
>>     care of removing deleted document
>>
>>
>>     On Oct 27, 2016 4:20 AM, "Arkadi Colson" <arkadi@smartbit.be
>>     <mailto:arkadi@smartbit.be>> wrote:
>>
>>         Hi
>>
>>         As you can see in the screenshot above in the oldest segments
>>         there are a lot of deletions. In total the shard has about
>>         26% deletions. How can I get rid of them so the index will be
>>         smaller again?
>>         Can this only be done with an optimize or does it also depend
>>         on the merge policy? If it also depends also on the merge
>>         policy which one should I choose then?
>>
>>         Thanks!
>>
>>         BR,
>>         Arkadi
>>
>
>


Mime
View raw message