zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Guillermo Vega-Toro" <gvega...@us.ibm.com>
Subject Re: How to add nodes to a Zookeeper 3.5.3-beta ensemble with reconfigEnabled=false
Date Fri, 23 Jun 2017 14:18:01 GMT
Thanks all for looking at this.

Here is what I've found to make ensemble config changes work with 
3.5.3-beta and reconfigEnabled=false:

1. All servers must be stopped.
2. In one server, make the desired changes to zoo.cfg (group, server.x, 
weight.x), and delete the dynamicConfigFile property.
3. Start the one server above. You will notice that when the server 
starts, no dynamic file is created, as if the config is "on hold".
4. In another server, make the same changes to zoo.cfg, and start the 
server. If there is no quorum, no dynamic file will be created.
5. Repeat step 4 for all servers

Once a quorum is reached, the proposed config changes are applied, and a 
dynamic config file will appear on the servers. If you run a 'config' 
command on the zkCLI, the desired configuration will show.

Also, once quorum is reached, further ensemble changes to zoo.cfg followed 
by restart of a single server are ignored. It is necessary to stop all 
servers and do the steps above to make any changes to the ensemble.

Thanks,

Alexander Shraer <shralex@gmail.com> wrote on 06/23/2017 01:20:47 AM:

> From: Alexander Shraer <shralex@gmail.com>
> To: user@zookeeper.apache.org
> Date: 06/23/2017 01:21 AM
> Subject: Re: How to add nodes to a Zookeeper 3.5.3-beta ensemble 
> with reconfigEnabled=false
> 
> I'm not sure it's necessary for backward compatibility since rolling
> restarts for config changes are not really an api the system provides.
> 
> I'd think the ACL control and admin only API should be sufficient for
> security and would prefer to get rid of the flag. But if you must have 
it,
> we have to prevent both in memory config updates (most important) and
> config file updates if reconfig is disabled. This sounds like a small
> change in quorumpeer, but perhaps I'm forgetting something.
> 
> Cheers
> Alex
> 
> 
> On Thu, Jun 22, 2017 at 11:06 PM Michael Han <hanm@cloudera.com> wrote:
> 
> > Hi Alex, thanks for clarification!
> >
> > It makes sense to me that users should use reconfig instead of rolling
> > upgrade moving forward. The only concern is backward compatibility now 
we
> > drop the old rolling upgrade support: since 3.5.x needs to be backward
> > compatible with 3.4.x [1], I think we probably need support rolling 
upgrade
> > for 3.5 branch.
> >
> > As for this flag - I believe it's there and set to false because 
reconfig
> > is a security sensitive feature and for such features user has to opt 
in
> > explicitly. Our security team here also has similar recommendations 
when I
> > talked with them about what this feature could do. There are also some
> > discussions around this flag / why it's there in ZOOKEEPER-2014.
> >
> > [1]
> > 
https://cwiki.apache.org/confluence/display/ZOOKEEPER/ReleaseManagement
> >
> >
> > On Thu, Jun 22, 2017 at 10:39 PM, Alexander Shraer <shralex@gmail.com>
> > wrote:
> >
> > > Hi Michael,
> > >
> > > The described behavior is the intended one - in 3.5 configuration is 
part
> > > of the synced state and is updated
> > > when the server syncs with the leader. The only rolling upgrade I 
tested
> > > was to upgrade the software version
> > > of the servers - this should still work. But I didn't try to support
> > > rolling upgrade for upgrading the configuration,
> > > since this should be done through reconfig.
> > >
> > > I'm still not sure what's the purpose of this flag btw. Why would 
someone
> > > want to do rolling restarts which are prone
> > > to inconsistencies and data loss, when they can use reconfig ?
> > >
> > > Alex
> > >
> > >
> > >
> > >
> > > On Thu, Jun 22, 2017 at 10:18 PM, Michael Han <hanm@cloudera.com> 
wrote:
> > >
> > > > reconfigEnabled only disables reconfig command when
> > > reconfigEnabled=false;
> > > > it does not disable the feature by mute all code paths of the 
reconfig
> > > > feature introduced in ZOOKEEPER-107. So regardless of the value of
> > > > reconfigEnabled,
> > > > 3.5.x ZK will create static config file and dynamic config file in 
any
> > > > cases.
> > > >
> > > > This might create a problem for users who want to do rolling 
upgrade
> > the
> > > > old way - because now the critical config information is not 
stored in
> > > > zoo.cfg anymore and modifying cfg.dynamic file manually will not 
work
> > > > because a reconfig needs to go through quorum processors. I think 
this
> > is
> > > > the problem described in the thread.
> > > >
> > > > Alex, is reconfig compatible with rolling upgrade? I don't find
> > anything
> > > > mentioned in ZOOKEEPER-107 about this. Currently I think the 
answer is
> > > no,
> > > > which means for 3.5.x the only way to change membership of cluster 
is
> > > > through reconfig. Could you confirm this conclusion? If that is 
the
> > case
> > > we
> > > > need patch the reconfigEnabled so it completely disable all code 
path
> > of
> > > > the reconfig feature to leave the static zoo.cfg intact.
> > > >
> > > >
> > > > On Thu, Jun 22, 2017 at 9:35 PM, Alexander Shraer 
<shralex@gmail.com>
> > > > wrote:
> > > >
> > > > > This sounds like a bug in the implementation of reconfigEnabled.
> > > > > Could you please open a JIRA with the description you provided ?
> > > > >
> > > > > Out of curiosity, why do you disable reconfig ? It is intended
> > exactly
> > > > > to perform the changes you're trying to make, in a simple and 
correct
> > > > way.
> > > > >
> > > > > Thanks,
> > > > > Alex
> > > > >
> > > > > On Thu, Jun 22, 2017 at 3:17 PM, Guillermo Vega-Toro <
> > > > gvegator@us.ibm.com>
> > > > > wrote:
> > > > >
> > > > > > I'm still unable to make configuration changes when
> > > > reconfigEnabled=false
> > > > > > by updating zoo.cfg and restarting the servers.
> > > > > >
> > > > > > For example, I want to change the weight of one of my servers.

I
> > edit
> > > > > > zoo.cfg on the server I want to change, and specify the group,
> > > > server.x,
> > > > > > and weight.x properties for all servers. I also remove the
> > > > > > dynamicConfigFile property and delete the dynamic config file.

I
> > then
> > > > > > restart the server. As soon as the server starts, the dynamic
> > config
> > > > file
> > > > > > re-appears, and it has the last configuration, as if the 
changes I
> > > made
> > > > > in
> > > > > > zoo.cfg were ignored. The dynamic configuration file on the

other
> > > > servers
> > > > > > also stays the same.
> > > > > >
> > > > > > What would be the correct way to achieve this (change a 
server's
> > > > weight,
> > > > > > or role) when reconfigEnabled=false and the CLI reconfig 
command
> > > cannot
> > > > > be
> > > > > > used?
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Cheers
> > > > Michael.
> > > >
> > >
> >
> >
> >
> > --
> > Cheers
> > Michael.
> >



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message