zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Shraer <shra...@gmail.com>
Subject Re: Dynamic reconfiguration: New participant server opens wrong ports
Date Wed, 25 Jun 2014 19:46:54 GMT
no problem. yes - the leader only counts acks from followers in the config.

Alex



On Wed, Jun 25, 2014 at 12:37 PM, Niko Vuokko <niko.vuokko@gmail.com> wrote:

> Thanks for the clarification,
>
> I've actually read that doc already. One more question then: how do I know
> whether a follower is of voting or non-voting kind? Is it simply equivalent
> to seeing or not seeing the follower as part of the quorum's configuration?
>
> niko
>
>
>
> 2014-06-25 13:27 GMT+03:00 Alexander Shraer <shralex@gmail.com>:
>
> > Hi Niko,
> >
> > Please see documentation here:
> > https://issues.apache.org/jira/browse/ZOOKEEPER-1660
> >
> > When adding servers to an ensemble, we use the "feature" where the leader
> > allows any server to
> > connect. After a new server connects and syncs with the leader one can
> > invoke the reconfig command
> > to actually make it part of the ensemble (the actual requirement is that
> a
> > quorum of the new config has to
> > be in sync with the leader when the reconfig is invoked, otherwise it
> will
> > fail).
> >
> > So yes - any server can connect as a non-voting follower. Its not part of
> > the ensemble. This was the case before dynamic reconfig support, but now
> > we're actually making sure it can't vote.
> >
> >
> > Alex
> >
> >
> >
> >
> >
> >
> > On Wed, Jun 25, 2014 at 12:40 PM, Niko Vuokko <niko.vuokko@gmail.com>
> > wrote:
> >
> > > Another reconfiguration weirdness question:
> > > Steps: Three servers in total, servers 1 and 2 know only about
> > themselves,
> > > server 3 knows everyone. I first start up 1 and 2. After quorum is
> > formed I
> > > start 3.
> > >
> > > Expected: server 3 cannot join or at most becomes an observer.
> > > Actually: server 3 is allowed in to the quorum as follower and I can do
> > > writes through it. The configuration, however, doesn't include server 3
> > > (4-letter conf or zkCli config) as all three servers list only 1 and 2.
> > >
> > > I think I saw some discussion about accepting unannounced new members,
> > but
> > > couldn't find it now. Even then, the new member obviously should be
> part
> > of
> > > the official configuration.
> > >
> > > BR,
> > > Niko Vuokko
> > >
> > >
> > >
> > > 2014-06-25 10:10 GMT+03:00 Niko Vuokko <niko.vuokko@gmail.com>:
> > >
> > > > JIRA+patch available @
> > > > https://issues.apache.org/jira/browse/ZOOKEEPER-1946
> > > >
> > > >
> > > > 2014-06-19 17:52 GMT+03:00 Alexander Shraer <shralex@gmail.com>:
> > > >
> > > > Hi,
> > > >>
> > > >> 1) yes. The only consistent way to change a configuration is to
> agree
> > > on a
> > > >> new one and currently you need to use the reconfig api to do that.
> > > >> 2) yeah seems confusing indeed. It probably makes sense to update
> the
> > > >> printed info when the config is updated (in this case when it learns
> > it
> > > >> during sync). Can you open a JIRA for this? If you can take a stab
> at
> > > >> fixing it I'll review it.
> > > >>
> > > >> Thanks
> > > >> Alex
> > > >> On Jun 19, 2014 4:52 PM, "Niko Vuokko" <niko.vuokko@gmail.com>
> wrote:
> > > >>
> > > >> > I basically try to simulate having a clean slate server coming
up
> > with
> > > >> > myid=3. So it would have no data, no dynamic configuration file
> > since
> > > >> the
> > > >> > ZK service has never run. I was hoping to test the reconfiguration
> > API
> > > >> to
> > > >> > let the new server 3 in to the quorum, but then this happened.
> > > >> >
> > > >> > "Static configuration" = $ZOOCFG on service start
> > > >> > Reconfiguration API used: Not yet...
> > > >> > Manual change: Yes, I kill server 3 manually, delete all its
data,
> > > >> change
> > > >> > its configuration and start it up as "new, empty server"
> > > >> >
> > > >> > Yes, the remaining quorum obviously tells the "new" server 3
to
> > > >> configure
> > > >> > itself with the old ports, but that just sounds *really* weird
to
> > me.
> > > >> Two
> > > >> > questions:
> > > >> >
> > > >> > 1) Is it really so that it doesn't matter how I configure the
new
> > > >> member if
> > > >> > the quorum already knows about an old server with the same myid?
> The
> > > old
> > > >> > configuration will just be forced upon the new server even if
that
> > is
> > > >> > unwanted.
> > > >> > 2) All the log entries refer to the new port 2184 although that
is
> > not
> > > >> > actually used. For example, once the "new" server 3 has joined
as
> a
> > > >> > follower, I'm still getting rows like
> > > >> >
> > > >> > 2014-06-19 16:25:10,883 [myid:3] - INFO
> > > >> > [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2184:QuorumPeer@972] -
> > FOLLOWING
> > > >> >
> > > >> > And as I mentioned, there is nothing in port 2184, the "new"
> server
> > 3
> > > >> > responds at 2183... The problematic piece seems to come from
> > > >> QuorumPeer@857
> > > >> > where the thread name is set. This bug causes confusion
> (especially
> > if
> > > >> > hostnames change as well), probably nothing worse in most cases
> > > though.
> > > >> >
> > > >> >
> > > >> > (btw, 4-letter word conf gives the currently known member
> > > configuration)
> > > >> >
> > > >> > Thanks for the help,
> > > >> > niko
> > > >> >
> > > >> >
> > > >> >
> > > >> > 2014-06-19 16:22 GMT+03:00 Alexander Shraer <shralex@gmail.com>:
> > > >> >
> > > >> > > when you say "change its static configuration", what exactly
do
> > you
> > > >> mean
> > > >> > ?
> > > >> > > this part of the configuration should be located in the
> membership
> > > >> file
> > > >> > > (dynamic part of the config). do you use the reconfiguration
API
> > to
> > > >> > change
> > > >> > > it (this is the right way) ? do you manually change it at
> all/some
> > > of
> > > >> the
> > > >> > > servers (this would have no effect because servers read
these
> > files
> > > >> only
> > > >> > > when they boot) ?
> > > >> > >
> > > >> > > In the scenario you describe I expect server 3 to come up.
Its
> > > >> possible
> > > >> > if
> > > >> > > you changed the config files manually that during sync with
the
> > > leader
> > > >> > the
> > > >> > > leader pushes the old config to server 3 that it has in
memory
> > > (since
> > > >> the
> > > >> > > files you updated are not read) that's why it effectively
has
> the
> > > old
> > > >> > > parameters.
> > > >> > >
> > > >> > > Please use the dynamic reconfig API when you make changes
to
> > > existing
> > > >> > > servers or add/remove servers.
> > > >> > >
> > > >> > > the "config" command in CLI can show you what config is
latest
> at
> > > the
> > > >> > > server you're connected too (if I remember correctly there's
> also
> > a
> > > 4
> > > >> > > letter command). You can also check the config file(s) of
> server 3
> > > >> after
> > > >> > it
> > > >> > > syncs with the leader. I suspect that it will contain the
old
> > > config.
> > > >> > >
> > > >> > >
> > > >> > > Alex
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > On Thu, Jun 19, 2014 at 2:03 PM, Niko Vuokko <
> > niko.vuokko@gmail.com
> > > >
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Starting from a stable 3-member quorum:
> > > >> > > >
> > > >> > > > server.1=localhost:2801:3801;2181
> > > >> > > > server.2=localhost:2802:3802;2182
> > > >> > > > server.3=localhost:2803:3803;2183
> > > >> > > >
> > > >> > > > I then kill server 3, clear its data directory, keep
its
> myid=3
> > > and
> > > >> > > change
> > > >> > > > its static configuration to
> > > >> > > >
> > > >> > > > server.1=localhost:2801:3801;2181
> > > >> > > > server.2=localhost:2802:3802;2182
> > > >> > > > server.3=localhost:2804:3804;2184
> > > >> > > >
> > > >> > > > Now what I would expect is that this "new" server 3
will not
> > join
> > > >> the
> > > >> > > > quorum since the ports don't match what the servers
1 and 2
> > > expect.
> > > >> > > > However, it can join. The problem is that the "new"
server 3
> > does
> > > >> not
> > > >> > > > respect its configuration. Its logs will contain the
new port
> > > number
> > > >> > > 2184,
> > > >> > > > but it will actually pick up the dynamic configuration
offered
> > by
> > > >> the
> > > >> > > > quorum and open up the old ports 2183 etc. After joining
> again,
> > > the
> > > >> > > dynamic
> > > >> > > > configuration file for server 3 contains
> > > >> > > >
> > > >> > > > server.3=localhost:2803:3803:participant;0.0.0.0:2183
> > > >> > > >
> > > >> > > > Also, echo conf | localhost 2184 never replies but
echo conf |
> > > >> > localhost
> > > >> > > > 2183 returns
> > > >> > > >
> > > >> > > > server.3=localhost:2803:3803:participant;0.0.0.0:2183
> > > >> > > >
> > > >> > > > Is this actually intentional or a bug?
> > > >> > > >
> > > >> > > >
> > > >> > > > Best,
> > > >> > > > Niko Vuokko
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message