lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <apa...@elyograg.org>
Subject Re: Solr Index upgradation Merging issue observed
Date Tue, 10 Jan 2017 04:11:37 GMT
On 1/8/2017 11:21 PM, Manan Sheth wrote:
> Currently, We are in process of upgrading existing Solr indexes from Solr 4.x to Solr
6.2.1. In order to upgrade existing indexes we are planning to use IndexUpgrader class in
sequential manner from Solr 4.x to Solr 5.x and Solr 5.x to Solr 6.2.1.
>
> While preforming the upgrdation a strange behaviour is noticed where all the previous
segments are getting merged to one single large segment. We need to preserve the original
segments as single large segment is getting bulkier (~ 2-4 TBs).
>
> Please let me know how to tune the process or write custom logic to overcome it.

I've taken a very quick look at the IndexUpgrader code.  It does the
upgrade by calling forceMerge ... which is the Lucene term for what Solr
still calls "optimize."  What you are seeing is completely normal.  This
is how it is designed to work.  Changing it *might* be possible, but it
would involve development work in Lucene.  It likely would not be quick
and easy.

Single-segment indexes are slightly faster than multi-segment indexes
containing the same data, so I am failing to see why this is a problem. 
After the upgrade, the total amount of disk space for your index would
either go down or stay the same, although it will temporarily double in
size during the upgrade.

FYI -- this kind of segment merging can happen as a consequence of
normal indexing, so this is something your system must be prepared to
handle even when you are not using the upgrader tool, on ANY version of
Solr.

Thanks,
Shawn


Mime
View raw message