zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Guillermo Vega-Toro" <gvega...@us.ibm.com>
Subject Re: How to add nodes to a Zookeeper 3.5.3-beta ensemble with reconfigEnabled=false
Date Fri, 23 Jun 2017 16:09:47 GMT
To summarize the issue:

1. A whole ensembe outage is required to modify the ensemble without the 
reconfig CLI (stop all servers, modify zoo.cfg, restart servers one by 
one).
2. Even with the reconfig CLI, if there is no quorum, it is not possible 
to re-configue the ensemble, so one has to fall back to modify the 
ensemble through modification of zoo.cfg and restart.

Are these things that can be fixed? 

#1 is a change in behavior from 3.4.6. We did not need to stop the whole 
ensemble in 3.4.6 to modify the ensemble configuration. We could restart 
servers one by one, and eventually a new quorum would be formed.

#2 may be an oversight in the implementation of reconfig. Consider the 
case where you have a 5-node ensemble, and 3 nodes go down. There is no 
quorum. So, it is not possible to use the reconfig command to for example, 
temporarily give a weight of 0 to the failed servers, and a weight of 1 to 
the surviving servers, which demotes the ensembe to just 2 nodes until the 
outage is resolved. It is necessary to fall back to re-configure the 
ensemble through editing zoo.cfg and restarting.

Thanks

"Guillermo Vega-Toro" <gvegator@us.ibm.com> wrote on 06/23/2017 09:18:01 
AM:

> From: "Guillermo Vega-Toro" <gvegator@us.ibm.com>
> To: user@zookeeper.apache.org
> Date: 06/23/2017 09:18 AM
> Subject: Re: How to add nodes to a Zookeeper 3.5.3-beta ensemble 
> with reconfigEnabled=false
> 
> Thanks all for looking at this.
> 
> Here is what I've found to make ensemble config changes work with 
> 3.5.3-beta and reconfigEnabled=false:
> 
> 1. All servers must be stopped.
> 2. In one server, make the desired changes to zoo.cfg (group, server.x, 
> weight.x), and delete the dynamicConfigFile property.
> 3. Start the one server above. You will notice that when the server 
> starts, no dynamic file is created, as if the config is "on hold".
> 4. In another server, make the same changes to zoo.cfg, and start the 
> server. If there is no quorum, no dynamic file will be created.
> 5. Repeat step 4 for all servers
> 
> Once a quorum is reached, the proposed config changes are applied, and a 

> dynamic config file will appear on the servers. If you run a 'config' 
> command on the zkCLI, the desired configuration will show.
> 
> Also, once quorum is reached, further ensemble changes to zoo.cfg 
followed 
> by restart of a single server are ignored. It is necessary to stop all 
> servers and do the steps above to make any changes to the ensemble.
> 
> Thanks,
> 
> Alexander Shraer <shralex@gmail.com> wrote on 06/23/2017 01:20:47 AM:
> 
> > From: Alexander Shraer <shralex@gmail.com>
> > To: user@zookeeper.apache.org
> > Date: 06/23/2017 01:21 AM
> > Subject: Re: How to add nodes to a Zookeeper 3.5.3-beta ensemble 
> > with reconfigEnabled=false
> > 
> > I'm not sure it's necessary for backward compatibility since rolling
> > restarts for config changes are not really an api the system provides.
> > 
> > I'd think the ACL control and admin only API should be sufficient for
> > security and would prefer to get rid of the flag. But if you must have 

> it,
> > we have to prevent both in memory config updates (most important) and
> > config file updates if reconfig is disabled. This sounds like a small
> > change in quorumpeer, but perhaps I'm forgetting something.
> > 
> > Cheers
> > Alex
> > 
> > 
> > On Thu, Jun 22, 2017 at 11:06 PM Michael Han <hanm@cloudera.com> 
wrote:
> > 
> > > Hi Alex, thanks for clarification!
> > >
> > > It makes sense to me that users should use reconfig instead of 
rolling
> > > upgrade moving forward. The only concern is backward compatibility 
now 
> we
> > > drop the old rolling upgrade support: since 3.5.x needs to be 
backward
> > > compatible with 3.4.x [1], I think we probably need support rolling 
> upgrade
> > > for 3.5 branch.
> > >
> > > As for this flag - I believe it's there and set to false because 
> reconfig
> > > is a security sensitive feature and for such features user has to 
opt 
> in
> > > explicitly. Our security team here also has similar recommendations 
> when I
> > > talked with them about what this feature could do. There are also 
some
> > > discussions around this flag / why it's there in ZOOKEEPER-2014.
> > >
> > > [1]
> > > 
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/ReleaseManagement
> > >
> > >
> > > On Thu, Jun 22, 2017 at 10:39 PM, Alexander Shraer 
<shralex@gmail.com>
> > > wrote:
> > >
> > > > Hi Michael,
> > > >
> > > > The described behavior is the intended one - in 3.5 configuration 
is 
> part
> > > > of the synced state and is updated
> > > > when the server syncs with the leader. The only rolling upgrade I 
> tested
> > > > was to upgrade the software version
> > > > of the servers - this should still work. But I didn't try to 
support
> > > > rolling upgrade for upgrading the configuration,
> > > > since this should be done through reconfig.
> > > >
> > > > I'm still not sure what's the purpose of this flag btw. Why would 
> someone
> > > > want to do rolling restarts which are prone
> > > > to inconsistencies and data loss, when they can use reconfig ?
> > > >
> > > > Alex
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Jun 22, 2017 at 10:18 PM, Michael Han <hanm@cloudera.com>

> wrote:
> > > >
> > > > > reconfigEnabled only disables reconfig command when
> > > > reconfigEnabled=false;
> > > > > it does not disable the feature by mute all code paths of the 
> reconfig
> > > > > feature introduced in ZOOKEEPER-107. So regardless of the value 
of
> > > > > reconfigEnabled,
> > > > > 3.5.x ZK will create static config file and dynamic config file 
in 
> any
> > > > > cases.
> > > > >
> > > > > This might create a problem for users who want to do rolling 
> upgrade
> > > the
> > > > > old way - because now the critical config information is not 
> stored in
> > > > > zoo.cfg anymore and modifying cfg.dynamic file manually will not


> work
> > > > > because a reconfig needs to go through quorum processors. I 
think 
> this
> > > is
> > > > > the problem described in the thread.
> > > > >
> > > > > Alex, is reconfig compatible with rolling upgrade? I don't find
> > > anything
> > > > > mentioned in ZOOKEEPER-107 about this. Currently I think the 
> answer is
> > > > no,
> > > > > which means for 3.5.x the only way to change membership of 
cluster 
> is
> > > > > through reconfig. Could you confirm this conclusion? If that is 
> the
> > > case
> > > > we
> > > > > need patch the reconfigEnabled so it completely disable all code


> path
> > > of
> > > > > the reconfig feature to leave the static zoo.cfg intact.
> > > > >
> > > > >
> > > > > On Thu, Jun 22, 2017 at 9:35 PM, Alexander Shraer 
> <shralex@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > This sounds like a bug in the implementation of 
reconfigEnabled.
> > > > > > Could you please open a JIRA with the description you provided

?
> > > > > >
> > > > > > Out of curiosity, why do you disable reconfig ? It is intended
> > > exactly
> > > > > > to perform the changes you're trying to make, in a simple and

> correct
> > > > > way.
> > > > > >
> > > > > > Thanks,
> > > > > > Alex
> > > > > >
> > > > > > On Thu, Jun 22, 2017 at 3:17 PM, Guillermo Vega-Toro <
> > > > > gvegator@us.ibm.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I'm still unable to make configuration changes when
> > > > > reconfigEnabled=false
> > > > > > > by updating zoo.cfg and restarting the servers.
> > > > > > >
> > > > > > > For example, I want to change the weight of one of my 
servers. 
> I
> > > edit
> > > > > > > zoo.cfg on the server I want to change, and specify the

group,
> > > > > server.x,
> > > > > > > and weight.x properties for all servers. I also remove
the
> > > > > > > dynamicConfigFile property and delete the dynamic config

file. 
> I
> > > then
> > > > > > > restart the server. As soon as the server starts, the 
dynamic
> > > config
> > > > > file
> > > > > > > re-appears, and it has the last configuration, as if the

> changes I
> > > > made
> > > > > > in
> > > > > > > zoo.cfg were ignored. The dynamic configuration file on
the 
> other
> > > > > servers
> > > > > > > also stays the same.
> > > > > > >
> > > > > > > What would be the correct way to achieve this (change a

> server's
> > > > > weight,
> > > > > > > or role) when reconfigEnabled=false and the CLI reconfig

> command
> > > > cannot
> > > > > > be
> > > > > > > used?
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Cheers
> > > > > Michael.
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Cheers
> > > Michael.
> > >
> 
> 



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message