hadoop-zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benjamin Reed <br...@yahoo-inc.com>
Subject Re: Weird ephemeral node issue
Date Tue, 17 Aug 2010 16:26:58 GMT
there are two things to keep in mind when thinking about this issue:

1) if a zk client is disconnected from the cluster, the client is 
essentially in limbo. because the client cannot talk to a server it 
cannot know if its session is still alive. it also cannot close its session.

2) the client only finds out about session expiration events when the 
client reconnects to the cluster. if zk tells a client that its session 
is expired, the ephemerals that correspond to that session will already 
be cleaned up.

one of the main design points about zk is that zk only gives correct 
information. if zk cannot give correct information, it basically says "i 
don't know". connection loss exceptions and disconnected states are 
basically "i don't know".

generally applications we design go into a "safe" mode, meaning they may 
serve reads but reject changes, when disconnected from zk and only kill 
themselves when they find out their session has expired.


ps - session information is replicated to all zk servers, so if a leader 
dies, all replicas know the sessions that are currently active and their 

On 08/16/2010 09:03 PM, Ted Dunning wrote:
> Ben or somebody else will have to repeat some of the detailed logic for
> this, but it has
> to do with the fact that you can't be sure what has happened during the
> network partition.
> One possibility is the one you describe, but another is that the partition
> happened because
> a majority of the ZK cluster lost power and you can't see the remaining
> nodes.  Those nodes
> will continue to serve any files in a read-only fashion.  If the partition
> involves you losing
> contact with the entire cluster at the same time a partition of the cluster
> into a quorum and
> a minority happens, then your ephemeral files could continue to exist at
> least until the breach
> in the cluster itself is healed.
> Suffice it to say that there are only a few strategies that leave you with a
> coherent picture
> of the universe.  Importantly, you shouldn't assume that the ephemerals will
> disappear at
> the same time as the session expiration event is delivered.
> On Mon, Aug 16, 2010 at 8:31 PM, Qing Yan<qingyan@gmail.com>  wrote:
>> Ouch, is this the current ZK behavior? This is unexpected, if the
>> client get partitioned from ZK cluster, he should
>> get notified and take some action(e.g. commit suicide) otherwise how
>> to tell a ephemeral node is really
>> up or down? Zombie can create synchronization nightmares..
>> On Mon, Aug 16, 2010 at 7:22 PM, Dave Wright<wrightd@gmail.com>  wrote:
>>> Another possible cause for this that I ran into recently with the c
>> client -
>>> you don't get the session expired notification until you are reconnected
>> to
>>> the quorum and it informs you the session is lost.  If you get
>> disconnected
>>> and can't reconnect you won't get the notification.  Personally I think
>> the
>>> client api should track the session expiration time locally and
>> information
>>> you once it's expired.
>>> On Aug 16, 2010 2:09 AM, "Qing Yan"<qingyan@gmail.com>  wrote:
>>> Hi Ted,
>>>   Do you mean GC problem can prevent delivery of SESSION EXPIRE event?
>>> Hum...so you have met this problem before?
>>> I didn't see any OOM though, will look into it more.
>>> On Mon, Aug 16, 2010 at 12:46 PM, Ted Dunning<ted.dunning@gmail.com>
>> wrote:
>>>> I am assuming that y...

View raw message