zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Young <you...@gmail.com>
Subject Re: Zookeeper node replacement for solr cloud steps
Date Thu, 09 Feb 2017 21:48:23 GMT
I've had some experience with similar scenarios.  As long as you can
maintain quorum through the ZooKeeper changes and the current leader is
defined within the Solr configuration, a single Solr restart should be
sufficient.

Something along these lines should work:
1. Add new ZooKeeper node to ZooKeeper configurations (but don't start the
new node)
2. Restart all follower nodes (except new one)
3. Restart leader
This should ensure the new leader remains within the existing Solr
configuration (if the new leader is the node you're about to remove,
restart it to force another election)

# Danger zone in steps 4-5 (see notes below)
4. Start new ZooKeeper node to enter the ensemble
5. Configure and restart Solr
6. Stop ZooKeeper node to be deleted
7. Configure and restart all ZooKeeper followers to remove node to be
deleted
8. Restart leader

The danger zone is the time frame between steps 4 and 6 where a new
unexpected leader election here could cause Solr trouble if the new leader
is not configured in Solr yet.

You can avoid this danger all together if you have enough ZooKeeper nodes.
If you have 5+ nodes, you should be able to swap steps 4 and 6.
e.g. If you have 5+ nodes, the ensemble can withstand 2 node failures.
This allows you to stop the ZooKeeper to be removed before restarting
Solr.  And then start the new node after the Solr restart to complete the
ensemble.

Of course, you should test veryify steps yourself before running this in
Production...

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message