zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Washko, Daniel" <dwas...@gannett.com>
Subject Re: Zookeeper node replacement for solr cloud steps
Date Fri, 10 Feb 2017 15:21:35 GMT
Thank you very much Eric. I appreciate the response.

-- 
Daniel S Washko
Solutions Architect



Phone: 757 667 1463
dwashko@gannett.comgannett.com <http://www.gannett.com/>

On 2/9/17, 4:48 PM, "Eric Young" <younge@gmail.com> wrote:

    I've had some experience with similar scenarios.  As long as you can
    maintain quorum through the ZooKeeper changes and the current leader is
    defined within the Solr configuration, a single Solr restart should be
    sufficient.
    
    Something along these lines should work:
    1. Add new ZooKeeper node to ZooKeeper configurations (but don't start the
    new node)
    2. Restart all follower nodes (except new one)
    3. Restart leader
    This should ensure the new leader remains within the existing Solr
    configuration (if the new leader is the node you're about to remove,
    restart it to force another election)
    
    # Danger zone in steps 4-5 (see notes below)
    4. Start new ZooKeeper node to enter the ensemble
    5. Configure and restart Solr
    6. Stop ZooKeeper node to be deleted
    7. Configure and restart all ZooKeeper followers to remove node to be
    deleted
    8. Restart leader
    
    The danger zone is the time frame between steps 4 and 6 where a new
    unexpected leader election here could cause Solr trouble if the new leader
    is not configured in Solr yet.
    
    You can avoid this danger all together if you have enough ZooKeeper nodes.
    If you have 5+ nodes, you should be able to swap steps 4 and 6.
    e.g. If you have 5+ nodes, the ensemble can withstand 2 node failures.
    This allows you to stop the ZooKeeper to be removed before restarting
    Solr.  And then start the new node after the Solr restart to complete the
    ensemble.
    
    Of course, you should test veryify steps yourself before running this in
    Production...
    

Mime
View raw message