zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Shraer <shra...@gmail.com>
Subject Re: Is the value of $MYID allowed to change across runs in an HA ZK deployment?
Date Tue, 06 Feb 2018 06:22:05 GMT
Hi Jay,

Perhaps it also depends on the restart? if the restart is done gradually,
for example a leader is in the middle of collecting votes when one of the
voters gets a new id and votes twice instead of once ? If the restart is a
barrier, where all servers are shut down and then restarted, this shouldn't

In 3.5, cluster membership is written into the ZK database as well as
configuration files, and contains server ids and parameters (ports, IPs,
etc). If ids change, it sounds like the membership
information may be wrong.

Perhaps there are also some implications on the security-related configs ?
Someone else may want to comment on these.

In general, changing ids doesn't feel like a very safe method to me...


On Mon, Feb 5, 2018 at 10:47 AM, <jay.taylor@oracle.com> wrote:

> Greetings Zookeepers,
> I'm investigating possible ways for Zookeeper to run safely on top of
> Kubernetes clusters.
> When the zookeeper containers come online, the value for $MYID is
> initially derived from the Kubernetes pod name.  All active pod names
> are guaranteed to be unique within the cluster at any given point in time.
> Example values:
>     - zookeeper-0
>     - zookeeper-1
>     - zookeeper-2
> and the formula for $MYID is ((the trailing number of the pod name) + 1):
>     - zookeeper-0 => $MYID=1
>     - zookeeper-1 => $MYID=2
>     - zookeeper-2 => $MYID=3
> The part I'm uncertain of is the relationship between $MYID and ensuring
> each zookeeper data set stays in sync with the rest of the cluster,
> particularly across container restarts.  Restarts can lead to Zookeeper
> data set being launched with a different value of $MYID compared with
> the previous run.  I.e., Zookeeper may have already run on any given
> data set in the past when the myid file contained a different value.
> Is it part of the mechanism used to ensure all follower members are in
> sync with the current leader?  It seems to me that if the leader (or
> followers) keep track of their peers via myid and it gets changed, there
> could be problems.
> Initial testing (without much load) has gone fine and things seem to
> work fine when launched with updated $MYID values.  I've also been
> perusing the ZK source code and inspecting how myid is used, and nothing
> stood out to indicate that this will lead to future problems.  However,
> experience dictates that with distributed systems the devil is often in
> nuanced details, so I'm hoping the experts out there may be able to shed
> light about the internal dependencies on the value of myid.
> Specific questions:
>     - Is myid relied on to never change, or does it only need to be
> unique within the cluster at any given time?
>     - What are the risks with changing myid in relation to ZK data set
> directories across runs?
> Your insights will be greatly appreciated!
> Kind regards,
> Jay Taylor

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message