zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jared Cantwell <jared.cantw...@gmail.com>
Subject Re: Dynamic reconfiguration
Date Sat, 28 Jul 2012 02:57:53 GMT
We are trying to remove the need for all admin intervention so that is one failure scenario
that is interesting to us. 

Jared

On Jul 27, 2012, at 7:42 PM, Alexander Shraer <shralex@gmail.com> wrote:

> Yes, this entry will be deleted. I don't like this either - if a new follower reboots
before added to the config it will not be able to boot up without manual help from an admin.
That's why I'm considering maybe to remove the check that a participant must always initially
be in its own config, but for now its there.
> 
> Alex
> 
> On Fri, Jul 27, 2012 at 6:34 PM, Jared Cantwell <jared.cantwell@gmail.com> wrote:
> Sorry for the confusion in terminology, I was unfamiliar with the exact leader/follower
semantics previously. 
> 
> So if all connected servers update their config file, does that mean that non-voting
followers who aren't part of the new ensemble will lose the entry specific to them in their
config file?  I can test this myself, but getting an inside perspective is very helpful. 
> 
> Thanks again for the help!
> Jared
> 
> 
> On Jul 27, 2012, at 6:55 PM, Alexander Shraer <shralex@gmail.com> wrote:
> 
>> Yes, any number of followers which are not in the configuration can just connect
and listen in. This has always been the case, also in 3.4, I just made use of this for the
purpose of adding members during reconfiguration. Moreover, in 3.4 there this bug ZOOKEEPER-1113
>> where the leader actually counts the votes of anyone connected, regardless of config
membership :) This is fixed in ZK-107, so they are really non-voting followers. 
>> 
>> >   I am assuming that's the case, and that it is a follower (and not 
>> > participant) by virtue of not being in the official configuration stored in

>> > zookeeper itself. 
>> 
>> Follower and participant types of servers is not something that was defined in ZK-107.
In ZooKeeper every follower/leader is a "participant". Its just that the votes of participants
that are not in the configuration are not counted that's why we call them non-voting followers.
BTW, obviously a non-voting follower can not become leader (like ZK-1113 this was also not
enforced before ZK-107).
>> 
>> > And a followup... does zookeeper only overwrite the dynamic 
>> > configuration file for nodes that are voting participants?  Such that if I 
>> > started a follower and then left it running through some 
>> > reconfigurations, its file would not get updated if it was never added as 
>> > part of those reconfigurations?
>> 
>> No, as soon as it connects to the current leader, its dynamic config file is overwritten
with the current configuration as part of the synchronization with the leader. Every time
a new configuration is committed, all connected servers (voting, non-voting, observers) will
update their dynamic config file, doesn't matter if they're in the config.
>> 
>> Alex
>> 
>> On Fri, Jul 27, 2012 at 5:35 PM, Jared Cantwell <jared.cantwell@gmail.com>
wrote:
>> So does just having the server started and pointing to the existing ensemble automatically
make it a "non participating follower"?  In other words, there is no need to inform the existing
nodes that this new node is joining as a follower?  And to extend that, there could be any
number of followers that are simply listening in on the event stream?  I am assuming that's
the case, and that it is a follower (and not participant) by virtue of not being in the official
configuration stored in zookeeper itself.
>> 
>> On Fri, Jul 27, 2012 at 6:29 PM, Alexander Shraer <shralex@gmail.com> wrote:
>> there are just two supported types - participant and observer.
>> (participant can act as either follower or leader).
>> 
>> So you can either write participant or leave it unspecified (which means participant
by default). Also, since the ip is the same for all your ports you don't have to write it
twice.  All of these should work in the same way:
>> 
>> server.5=10.10.5.17:2182:2183:participant;10.10.5.17:2181
>> server.5=10.10.5.17:2182:2183:participant;2181
>> server.5=10.10.5.17:2182:2183;10.10.5.17:2181
>> server.5=10.10.5.17:2182:2183;2181
>> 
>> 
>> 
>> On Fri, Jul 27, 2012 at 5:25 PM, Jared Cantwell <jared.cantwell@gmail.com>
wrote:
>> Thanks Alex for the response.  Our current lines in the configuration look like this:
>> 
>> server.5=10.10.5.17:2182:2183:participant;10.10.5.17:2181
>> 
>> For the new servers is it ok for their entry to have "participant"?  Or should that
be something different (e.g. "follower")?
>> 
>> ~Jared
>> 
>> On Fri, Jul 27, 2012 at 6:20 PM, Alexander Shraer <shralex@gmail.com> wrote:
>> Hi Jared,
>> 
>> Thanks for experimenting with this feature. 
>> 
>> The idea is that new servers join as "non voting followers". Which means that they
act as normal followers but the leader ignores their votes since they are not part of the
current configuration. The leader only counts their votes during the reconfiguration itself
(to make sure a quorum of the new config is ready before the new config can be committed/activated).
Defining them as observers is not a good idea, for example in your scenario if they were observers
they wouldn't be able to participate in the reconfiguration protocol (which is similar to
the protocol for committing any other operation in which observers don't participate) and
since we don't have a quorum of followers in the new config that can ack, reconfiguration
would throw an exception (of KeeperException.NEWCONFIGNOQUORUM type). 
>> Of course if you intend them to be observers in the new config you can define them
as observers since their votes are not needed during reconfig anyway.
>> 
>> You're right, the new servers must be able to connect to the old quorum. At minimum,
their file should contain the current leader, but 
>> you can also copy the current configuration file to the new members if you wish.

>> 
>> In addition, you should add a line for the member itself, so that server F appears
in F's config file (Its not important that the other new servers appear in F's file, but it
won't hurt either, so you can do a union of old and new if you wish). The constructor of QuorumPeer
checks that the server itself is in the configuration its started with, otherwise its not
going to run. This check has always been there, but I'm thinking of possibly changing it in
the future.
>> 
>> As soon as F connects to the leader, its config file will be overwritten with the
current config file as part of the synchronization process. 
>> 
>> Alex
>> 
>> 
>> On Fri, Jul 27, 2012 at 10:06 AM, Jared Cantwell <jared.cantwell@gmail.com>
wrote:
>> Hi,
>> 
>> We are testing integration with 3.5.0 and dynamic membership and I have a
>> question.  If I have a current set of servers in my ensemble {A,B,C,D,E}
>> and I want to reconfigure the ensemble to {D,E,F,G,H}, how should the
>> dynamic config file on servers F,G,H be configured on startup?  Should they
>> have the old ensemble, the new ensemble, or the union of both ensembles?
>>  It seems like these new servers need to  know about the old quorum, but
>> since they aren't part of it yet its not clear to me how they should be
>> configured.  Should there be an intermediate configuration with F,G, and H
>> as simply Observers?
>> 
>> I can't find much documentation on this so I want to make sure I understand
>> things correctly.
>> 
>> Thanks!
>> ~Jared
>> 
>> 
>> 
>> 
>> 
> 

Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message