lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Moving to HDFS, How to merge indices from 8 servers ?‏‏
Date Thu, 11 Sep 2014 21:41:48 GMT
Um, I really think this is pretty likely to not be a great solution.
When you say "merge indexes", I'm thinking you want to go from 8
shards to 1 shard. Now, this can be done with the "merge indexes" core
admin API, see:
https://wiki.apache.org/solr/MergingSolrIndexes

BUT.
1>  This will break all things SolrCloud-ish assuming you created your
8 shards under SolrCloud.
2> Solr is usually limited by memory, so trying to fit enough of your
single huge index into memory may be problematical.

This feels like an XY problem, _why_ are you asking about this? What
is the use-case you want to handle by this?

Best,
Erick

On Thu, Sep 11, 2014 at 7:44 AM, Amey Jadiye
<ameyjadiye@codeinventory.com> wrote:
> FYI, I searched the google for this problem but didn't find any satisfactory answer.Here
is the current situation : I have the 8 shards in my solr cloud backed up with 3 zookeeper
all are setup on AWS EC2 instances, all 8 are leader with no replicas.I have only 1 collection
say collection1 divided in 8 shards, i have configured the index and tlog folder on each server
pointing into 1TB EBS disk attached to each servers, all 8 servers are having around 100GB
for index folder each. so total index files i have is ~800Gb.Now, i want to move all the data
to HDFS, so I am going to setup the HDFS on all 8 serversMerge all the indexes from 8 serversPut
in HDFS.Stop  and Start my all solr servers on HDFS to access that common index data with
setting  below cp parameter and few more.-Dsolr.directoryFactory=HdfsDirectoryFactory    
-Dsolr.lock.type=hdfs     -Dsolr.data.dir=hdfs://host:port/path     -Dsolr.updatelog=hdfs://host:port/path
-jarNow could you tell me is this correct approach? if yes how can i merge all indices from
8 server ?Regards,Amey

Mime
View raw message