lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emir Arnautović <emir.arnauto...@sematext.com>
Subject Re: Merging of index in Solr
Date Tue, 21 Nov 2017 08:52:13 GMT
Hi Edwin,
How many host/nodes/shard are those 3.5TB? I am not familiar with merge code, but trying to
think what it might include, so don’t take any of following as ground truth.
Merging for sure will include segments rewrite, so you better have additional 3.5TB if you
are merging it to a single segment. But that should not last days on SSD. My guess is that
you are running on the edge of your heap and doing a lot GCs and maybe you will OOM at some
point. I would guess that merging is memory intensive operation and even if not holding large
structures in memory, it will probably create a lot of garbage. Merging requires a lot of
comparison so it is also a possibility that you are exhausting CPU resources.
Bottom line - without more details and some monitoring tool, it is hard to tell why it is
taking that much.
And there is also a question if merging is good choice in you case - is index static/updates
free?

Regards,
Emir 
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 20 Nov 2017, at 17:35, Zheng Lin Edwin Yeo <edwinyeozl@gmail.com> wrote:
> 
> Hi,
> 
> Does anyone knows how long usually the merging in Solr will take?
> 
> I am currently merging about 3.5TB of data, and it has been running for
> more than 28 hours and it is not completed yet. The merging is running on
> SSD disk.
> 
> I am using Solr 6.5.1.
> 
> Regards,
> Edwin


Mime
View raw message