Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Subject: Re: Merging of index in Solr
To: solr-user@lucene.apache.org
References: <CAF2DzVUnd-fWFpCXGMfXLgRM-wBgFoche6CPDNABEX5nvtUP0g@mail.gmail.com>
 <fb38027c-7921-8b47-fbf0-dd1dfa478ff5@elyograg.org>
 <CAF2DzVX8wDqA2+FqUvbbYaPrtKmJyutPoV7mwMjyGXDHyMkphg@mail.gmail.com>
 <a7845c16-729b-31dd-a272-6ce35e091e49@elyograg.org>
 <CAN4YXvcavwd5Rm28X0fe_xOWHF=uOVL7bt-qLBMcq8r8aKvbvw@mail.gmail.com>
 <CAF2DzVUr_vD_vmic8WRr8t2VY88yD-Y0Vb5b6XQ_bqJ=6QJ8eA@mail.gmail.com>
From: Shawn Heisey <apache@elyograg.org>
Message-ID: <9e375b99-3643-a825-bfae-6719790c1a1c@elyograg.org>
Date: Wed, 22 Nov 2017 21:50:56 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101
 Thunderbird/52.4.0
MIME-Version: 1.0
In-Reply-To: <CAF2DzVUr_vD_vmic8WRr8t2VY88yD-Y0Vb5b6XQ_bqJ=6QJ8eA@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Content-Language: en-US
archived-at: Thu, 23 Nov 2017 04:51:14 -0000

On 11/22/2017 6:19 PM, Zheng Lin Edwin Yeo wrote:
> I'm doing the merging on the SSD drive, the speed should be ok?

The speed of virtually all modern disks will have almost no influence on 
the speed of the merge.  The bottleneck isn't disk transfer speed, it's 
the operation of the merge code in Lucene.

As I said earlier in this thread, a merge is **NOT** just a copy. Lucene 
must completely rebuild the data structures of the index to incorporate 
all of the segments of the source indexes into a single segment in the 
target index, while simultaneously *excluding* information from 
documents that have been deleted.

The best speed I have ever personally seen for a merge is 30 megabytes 
per second.  This is far below the sustained transfer rate of a typical 
modern SATA disk.  SSD is capable of far faster data transfer ...but it 
will NOT make merges go any faster.

> We need to merge because the data are indexed in two different collections,
> and we need them to be under the same collection, so that we can do things
> like faceting more accurately.
> Will sharding alone achieve this? Or do we have to merge first before we do
> the sharding?

If you want the final index to be sharded, it's typically best to index 
from scratch into a new empty collection that has the number of shards 
you want.  The merging tool you're using isn't aware of concepts like 
shards.  It combines everything into a single index.

It's not entirely clear what you're asking with the question about 
sharding alone.  Making a guess:  I have never heard of facet accuracy 
being affected by whether or not the index is sharded.  If that *is* 
possible, then I would expect an index that is NOT sharded to have 
better accuracy.

Thanks,
Shawn