lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emir Arnautovic <emir.arnauto...@sematext.com>
Subject Re: Merge policy
Date Fri, 28 Oct 2016 08:25:19 GMT
I got some notification from mailer, so not sure if my reply reached you:

"If you are using TieredMergePolicy, you can try setting 
/*reclaimDeletesWeight*/."

HTH,
Emir


On 28.10.2016 09:20, Arkadi Colson wrote:
>
> The index size of 1 shard is about 125GB and we are running 11 shards 
> with replication factor 2 so it's a lot of data. The deletions 
> percentage at the bottom of the segment page is around 25%. So it's 
> quite some space which we could recover. That's why I was looking for 
> an optimize.
>
> Do you have any idea why the merge policy does not merge away the 
> deletions? Should I tweak some parameters somehow? It's a default 
> installation using the default settings and parameters. If you need 
> more info, just let me know...
>
> Thx!
>
>
> On 27-10-16 17:40, Erick Erickson wrote:
>> Why do you think you need to get rid of the deleted data? During normal
>> indexing, these will be "merged away". Optimizing has some downsides
>> for continually changing indexes, in particular since the default
>> tieredmergepolicy tries to merge "like size" segments, deletions will
>> accumulate in your one large segment and the percentage of
>> deleted documents may get even higher.
>>
>> Unless there's some measurable performance gain that the users
>> will notice, I'd just leave this alone.
>>
>> The exception here is if you have, say, an index that changes rarely
>> in which case optimizing then makes more sense.
>>
>> Best,
>> Erick
>>
>> On Thu, Oct 27, 2016 at 6:56 AM, Arkadi Colson <arkadi@smartbit.be 
>> <mailto:arkadi@smartbit.be>> wrote:
>>
>>     Thanks for the answer!
>>     Do you know if there is a way to trigger an optimize for only 1
>>     shard and not the whole collection at once?
>>
>>
>>     On 27-10-16 15:30, Pushkar Raste wrote:
>>>
>>>     Try commit with expungeDeletes="true"
>>>
>>>     I am not sure if it will merge old segments that have deleted
>>>     documents.
>>>
>>>     In the worst case you can 'optimize' your index which should
>>>     take care of removing deleted document
>>>
>>>
>>>     On Oct 27, 2016 4:20 AM, "Arkadi Colson" <arkadi@smartbit.be
>>>     <mailto:arkadi@smartbit.be>> wrote:
>>>
>>>         Hi
>>>
>>>         As you can see in the screenshot above in the oldest
>>>         segments there are a lot of deletions. In total the shard
>>>         has about 26% deletions. How can I get rid of them so the
>>>         index will be smaller again?
>>>         Can this only be done with an optimize or does it also
>>>         depend on the merge policy? If it also depends also on the
>>>         merge policy which one should I choose then?
>>>
>>>         Thanks!
>>>
>>>         BR,
>>>         Arkadi
>>>
>>
>>
>

-- 
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/


Mime
View raw message