zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Han <h...@cloudera.com>
Subject Re: How to add nodes to a Zookeeper 3.5.3-beta ensemble with reconfigEnabled=false
Date Fri, 23 Jun 2017 16:12:35 GMT
Guillermo, thank you for reporting the issue and share you findings on the
workaround.

I filed https://issues.apache.org/jira/browse/ZOOKEEPER-2819 which should
provide expected behavior - when reconfigEnabled=false you should be able
to do rolling restarts the old ways, once that JIRA is fixed.


On Fri, Jun 23, 2017 at 7:18 AM, Guillermo Vega-Toro <gvegator@us.ibm.com>
wrote:

> Thanks all for looking at this.
>
> Here is what I've found to make ensemble config changes work with
> 3.5.3-beta and reconfigEnabled=false:
>
> 1. All servers must be stopped.
> 2. In one server, make the desired changes to zoo.cfg (group, server.x,
> weight.x), and delete the dynamicConfigFile property.
> 3. Start the one server above. You will notice that when the server
> starts, no dynamic file is created, as if the config is "on hold".
> 4. In another server, make the same changes to zoo.cfg, and start the
> server. If there is no quorum, no dynamic file will be created.
> 5. Repeat step 4 for all servers
>
> Once a quorum is reached, the proposed config changes are applied, and a
> dynamic config file will appear on the servers. If you run a 'config'
> command on the zkCLI, the desired configuration will show.
>
> Also, once quorum is reached, further ensemble changes to zoo.cfg followed
> by restart of a single server are ignored. It is necessary to stop all
> servers and do the steps above to make any changes to the ensemble.
>
> Thanks,
>
> Alexander Shraer <shralex@gmail.com> wrote on 06/23/2017 01:20:47 AM:
>
> > From: Alexander Shraer <shralex@gmail.com>
> > To: user@zookeeper.apache.org
> > Date: 06/23/2017 01:21 AM
> > Subject: Re: How to add nodes to a Zookeeper 3.5.3-beta ensemble
> > with reconfigEnabled=false
> >
> > I'm not sure it's necessary for backward compatibility since rolling
> > restarts for config changes are not really an api the system provides.
> >
> > I'd think the ACL control and admin only API should be sufficient for
> > security and would prefer to get rid of the flag. But if you must have
> it,
> > we have to prevent both in memory config updates (most important) and
> > config file updates if reconfig is disabled. This sounds like a small
> > change in quorumpeer, but perhaps I'm forgetting something.
> >
> > Cheers
> > Alex
> >
> >
> > On Thu, Jun 22, 2017 at 11:06 PM Michael Han <hanm@cloudera.com> wrote:
> >
> > > Hi Alex, thanks for clarification!
> > >
> > > It makes sense to me that users should use reconfig instead of rolling
> > > upgrade moving forward. The only concern is backward compatibility now
> we
> > > drop the old rolling upgrade support: since 3.5.x needs to be backward
> > > compatible with 3.4.x [1], I think we probably need support rolling
> upgrade
> > > for 3.5 branch.
> > >
> > > As for this flag - I believe it's there and set to false because
> reconfig
> > > is a security sensitive feature and for such features user has to opt
> in
> > > explicitly. Our security team here also has similar recommendations
> when I
> > > talked with them about what this feature could do. There are also some
> > > discussions around this flag / why it's there in ZOOKEEPER-2014.
> > >
> > > [1]
> > >
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/ReleaseManagement
> > >
> > >
> > > On Thu, Jun 22, 2017 at 10:39 PM, Alexander Shraer <shralex@gmail.com>
> > > wrote:
> > >
> > > > Hi Michael,
> > > >
> > > > The described behavior is the intended one - in 3.5 configuration is
> part
> > > > of the synced state and is updated
> > > > when the server syncs with the leader. The only rolling upgrade I
> tested
> > > > was to upgrade the software version
> > > > of the servers - this should still work. But I didn't try to support
> > > > rolling upgrade for upgrading the configuration,
> > > > since this should be done through reconfig.
> > > >
> > > > I'm still not sure what's the purpose of this flag btw. Why would
> someone
> > > > want to do rolling restarts which are prone
> > > > to inconsistencies and data loss, when they can use reconfig ?
> > > >
> > > > Alex
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Jun 22, 2017 at 10:18 PM, Michael Han <hanm@cloudera.com>
> wrote:
> > > >
> > > > > reconfigEnabled only disables reconfig command when
> > > > reconfigEnabled=false;
> > > > > it does not disable the feature by mute all code paths of the
> reconfig
> > > > > feature introduced in ZOOKEEPER-107. So regardless of the value of
> > > > > reconfigEnabled,
> > > > > 3.5.x ZK will create static config file and dynamic config file in
> any
> > > > > cases.
> > > > >
> > > > > This might create a problem for users who want to do rolling
> upgrade
> > > the
> > > > > old way - because now the critical config information is not
> stored in
> > > > > zoo.cfg anymore and modifying cfg.dynamic file manually will not
> work
> > > > > because a reconfig needs to go through quorum processors. I think
> this
> > > is
> > > > > the problem described in the thread.
> > > > >
> > > > > Alex, is reconfig compatible with rolling upgrade? I don't find
> > > anything
> > > > > mentioned in ZOOKEEPER-107 about this. Currently I think the
> answer is
> > > > no,
> > > > > which means for 3.5.x the only way to change membership of cluster
> is
> > > > > through reconfig. Could you confirm this conclusion? If that is
> the
> > > case
> > > > we
> > > > > need patch the reconfigEnabled so it completely disable all code
> path
> > > of
> > > > > the reconfig feature to leave the static zoo.cfg intact.
> > > > >
> > > > >
> > > > > On Thu, Jun 22, 2017 at 9:35 PM, Alexander Shraer
> <shralex@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > This sounds like a bug in the implementation of reconfigEnabled.
> > > > > > Could you please open a JIRA with the description you provided
?
> > > > > >
> > > > > > Out of curiosity, why do you disable reconfig ? It is intended
> > > exactly
> > > > > > to perform the changes you're trying to make, in a simple and
> correct
> > > > > way.
> > > > > >
> > > > > > Thanks,
> > > > > > Alex
> > > > > >
> > > > > > On Thu, Jun 22, 2017 at 3:17 PM, Guillermo Vega-Toro <
> > > > > gvegator@us.ibm.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I'm still unable to make configuration changes when
> > > > > reconfigEnabled=false
> > > > > > > by updating zoo.cfg and restarting the servers.
> > > > > > >
> > > > > > > For example, I want to change the weight of one of my servers.
> I
> > > edit
> > > > > > > zoo.cfg on the server I want to change, and specify the
group,
> > > > > server.x,
> > > > > > > and weight.x properties for all servers. I also remove
the
> > > > > > > dynamicConfigFile property and delete the dynamic config
file.
> I
> > > then
> > > > > > > restart the server. As soon as the server starts, the dynamic
> > > config
> > > > > file
> > > > > > > re-appears, and it has the last configuration, as if the
> changes I
> > > > made
> > > > > > in
> > > > > > > zoo.cfg were ignored. The dynamic configuration file on
the
> other
> > > > > servers
> > > > > > > also stays the same.
> > > > > > >
> > > > > > > What would be the correct way to achieve this (change a
> server's
> > > > > weight,
> > > > > > > or role) when reconfigEnabled=false and the CLI reconfig
> command
> > > > cannot
> > > > > > be
> > > > > > > used?
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Cheers
> > > > > Michael.
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Cheers
> > > Michael.
> > >
>
>
>


-- 
Cheers
Michael.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message