zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Shraer <shra...@gmail.com>
Subject Re: Dynamic reconfiguration: New participant server opens wrong ports
Date Thu, 19 Jun 2014 14:52:38 GMT
Hi,

1) yes. The only consistent way to change a configuration is to agree on a
new one and currently you need to use the reconfig api to do that.
2) yeah seems confusing indeed. It probably makes sense to update the
printed info when the config is updated (in this case when it learns it
during sync). Can you open a JIRA for this? If you can take a stab at
fixing it I'll review it.

Thanks
Alex
On Jun 19, 2014 4:52 PM, "Niko Vuokko" <niko.vuokko@gmail.com> wrote:

> I basically try to simulate having a clean slate server coming up with
> myid=3. So it would have no data, no dynamic configuration file since the
> ZK service has never run. I was hoping to test the reconfiguration API to
> let the new server 3 in to the quorum, but then this happened.
>
> "Static configuration" = $ZOOCFG on service start
> Reconfiguration API used: Not yet...
> Manual change: Yes, I kill server 3 manually, delete all its data, change
> its configuration and start it up as "new, empty server"
>
> Yes, the remaining quorum obviously tells the "new" server 3 to configure
> itself with the old ports, but that just sounds *really* weird to me. Two
> questions:
>
> 1) Is it really so that it doesn't matter how I configure the new member if
> the quorum already knows about an old server with the same myid? The old
> configuration will just be forced upon the new server even if that is
> unwanted.
> 2) All the log entries refer to the new port 2184 although that is not
> actually used. For example, once the "new" server 3 has joined as a
> follower, I'm still getting rows like
>
> 2014-06-19 16:25:10,883 [myid:3] - INFO
> [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2184:QuorumPeer@972] - FOLLOWING
>
> And as I mentioned, there is nothing in port 2184, the "new" server 3
> responds at 2183... The problematic piece seems to come from QuorumPeer@857
> where the thread name is set. This bug causes confusion (especially if
> hostnames change as well), probably nothing worse in most cases though.
>
>
> (btw, 4-letter word conf gives the currently known member configuration)
>
> Thanks for the help,
> niko
>
>
>
> 2014-06-19 16:22 GMT+03:00 Alexander Shraer <shralex@gmail.com>:
>
> > when you say "change its static configuration", what exactly do you mean
> ?
> > this part of the configuration should be located in the membership file
> > (dynamic part of the config). do you use the reconfiguration API to
> change
> > it (this is the right way) ? do you manually change it at all/some of the
> > servers (this would have no effect because servers read these files only
> > when they boot) ?
> >
> > In the scenario you describe I expect server 3 to come up. Its possible
> if
> > you changed the config files manually that during sync with the leader
> the
> > leader pushes the old config to server 3 that it has in memory (since the
> > files you updated are not read) that's why it effectively has the old
> > parameters.
> >
> > Please use the dynamic reconfig API when you make changes to existing
> > servers or add/remove servers.
> >
> > the "config" command in CLI can show you what config is latest at the
> > server you're connected too (if I remember correctly there's also a 4
> > letter command). You can also check the config file(s) of server 3 after
> it
> > syncs with the leader. I suspect that it will contain the old config.
> >
> >
> > Alex
> >
> >
> >
> >
> > On Thu, Jun 19, 2014 at 2:03 PM, Niko Vuokko <niko.vuokko@gmail.com>
> > wrote:
> >
> > > Starting from a stable 3-member quorum:
> > >
> > > server.1=localhost:2801:3801;2181
> > > server.2=localhost:2802:3802;2182
> > > server.3=localhost:2803:3803;2183
> > >
> > > I then kill server 3, clear its data directory, keep its myid=3 and
> > change
> > > its static configuration to
> > >
> > > server.1=localhost:2801:3801;2181
> > > server.2=localhost:2802:3802;2182
> > > server.3=localhost:2804:3804;2184
> > >
> > > Now what I would expect is that this "new" server 3 will not join the
> > > quorum since the ports don't match what the servers 1 and 2 expect.
> > > However, it can join. The problem is that the "new" server 3 does not
> > > respect its configuration. Its logs will contain the new port number
> > 2184,
> > > but it will actually pick up the dynamic configuration offered by the
> > > quorum and open up the old ports 2183 etc. After joining again, the
> > dynamic
> > > configuration file for server 3 contains
> > >
> > > server.3=localhost:2803:3803:participant;0.0.0.0:2183
> > >
> > > Also, echo conf | localhost 2184 never replies but echo conf |
> localhost
> > > 2183 returns
> > >
> > > server.3=localhost:2803:3803:participant;0.0.0.0:2183
> > >
> > > Is this actually intentional or a bug?
> > >
> > >
> > > Best,
> > > Niko Vuokko
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message