lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Damien Dykman <damien.dyk...@gmail.com>
Subject Re: Transparently rebalancing a Solr cluster without splitting or moving shards
Date Tue, 08 Jul 2014 16:57:13 GMT
Thanks for your suggestions and recommendations.

If I understand correctly, the MIGRATE command does shard splitting 
(around the range of the split.key) and merging behind the scene. 
Though, it's a bit difficult to properly monitor the actual migration, 
set the proper timeouts, know when to direct indexing and search traffic 
to the destination collection, etc.

Note sure how to MIGRATE an entire collection. By providing the full 
list of split.keys? I'd be surprised if that was doable, but I guess it 
will skip the splitting part, which makes it easier ;-) Or much tougher 
by splitting around all the ranges. More seriously, doing a MERGEINDEX 
at the core level might not be a bad alternative, providing the hash 
ranges are compatible.

Damien

On 07/07/2014 05:14 PM, Shawn Heisey wrote:
>> I don't think you'd want to disable mmap. It could be done, by choosing
>> another DirectoryFactory object. Adding memory is likely to be the only
>> sane way forward.
>>
>> Another possibility would be to bump up the maxShardsPerNode value and
>> build the new collection (with the proper number of shards) only on the
>> new machines... Then when they are built, move them to their proper homes
>> and manually adjust the cluster state in zookeeper. This will still
>> generate a lot of I/O, but hopefully it will last for less time on the
>> wall clock, and it will be something you can do when load is low.
>>
>> After that done and you've switched to it, you can add replicas with
>> either the addreplica collections api or with the core admin api. You
>> should be on the newest Solr version... Lots of bugs have been found and
>> fixed.
>>
>> One thing I wonder is whether the MIGRATE api can be used on an entire
>> collection. It says it works by shard key, but I suspect that most users
>> will not be using that functionality.
>>
>> Thanks,
>> Shawn
>>

Mime
View raw message