Return-Path: X-Original-To: apmail-curator-user-archive@minotaur.apache.org Delivered-To: apmail-curator-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0C6E9100AA for ; Wed, 19 Mar 2014 13:00:48 +0000 (UTC) Received: (qmail 43653 invoked by uid 500); 19 Mar 2014 13:00:44 -0000 Delivered-To: apmail-curator-user-archive@curator.apache.org Received: (qmail 43561 invoked by uid 500); 19 Mar 2014 13:00:42 -0000 Mailing-List: contact user-help@curator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@curator.apache.org Delivered-To: mailing list user@curator.apache.org Received: (qmail 43535 invoked by uid 99); 19 Mar 2014 13:00:41 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Mar 2014 13:00:41 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of Matthew.Brown@citrix.com designates 66.165.176.63 as permitted sender) Received: from [66.165.176.63] (HELO SMTP02.CITRIX.COM) (66.165.176.63) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Mar 2014 13:00:35 +0000 X-IronPort-AV: E=Sophos;i="4.97,685,1389744000"; d="scan'208,217";a="111456222" Received: from sjcpex01cl02.citrite.net ([10.216.14.144]) by FTLPIPO02.CITRIX.COM with ESMTP/TLS/AES128-SHA; 19 Mar 2014 13:00:14 +0000 Received: from SJCPEX01CL01.citrite.net ([169.254.1.167]) by SJCPEX01CL02.citrite.net ([10.216.14.144]) with mapi id 14.02.0342.004; Wed, 19 Mar 2014 06:00:12 -0700 From: Matt Brown To: "user@curator.apache.org" , Robert Kamphuis Subject: Re: Confused about the LeaderLatch - what should happen on ConnectionState.SUSPENDED and ConnectionState.LOST ? Thread-Topic: Confused about the LeaderLatch - what should happen on ConnectionState.SUSPENDED and ConnectionState.LOST ? Thread-Index: AQHPQ02nBaPebgwBQkqWz/L4p+Avx5rokfsA Date: Wed, 19 Mar 2014 13:00:11 +0000 Message-ID: References: <2BF68D8D-7BFB-42C7-AC62-8079539BFBCF@supercell.com> <300227C1-7D23-48D7-8B4A-F86FBC51DDA5@supercell.com> In-Reply-To: <300227C1-7D23-48D7-8B4A-F86FBC51DDA5@supercell.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.3.9.131030 x-originating-ip: [10.13.107.79] Content-Type: multipart/alternative; boundary="_000_CF4F0BAB4F4Bmatthewbrowncitrixcom_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_CF4F0BAB4F4Bmatthewbrowncitrixcom_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable > My assumption and desired behaviour is that the user should suspend opera= tions - which implies to me that its leadership status is uncertain. (I am = holding off all persistent operations for example). > But -I think- this also implies that no-one else can become leader yet - = we either have the old-leader still be leader, and no one else, or then the= old-leader disappeared and we are in effect leaderless for some time. I think the second part of this is incorrect =96 if client 1 has lost it's = zookeeper connection, it doesn't imply that other clients have also lost th= eir zookeeper connection. So it would be correct for the former leader who now has a suspended connec= tion to cease it's leader activities =96 but other clients who are still co= nnected to the ensemble may have become the leader due to the suspension of= client 1's connection. If client 1 still acted as if it still might be the leader when it's connec= tion becomes suspended, then you would have two leaders =96 client 1 and wh= atever client which that still has a healthy ZK connection which grabbed th= e latch. >From the perspective of the zookeeper ensemble, it can't know if your clien= t is suffering from a "short connection break" or if it has died altogether= =96 so the client's leader role should be treated as lost in either case. From: Robert Kamphuis > Reply-To: "user@curator.apache.org" > Date: Wednesday, March 19, 2014 at 6:18 AM To: "user@curator.apache.org" > Cc: Robert Kamphuis > Subject: Confused about the LeaderLatch - what should happen on ConnectionS= tate.SUSPENDED and ConnectionState.LOST ? Hi, I have been working on changing our application to work with Zookeeper and = Curator for some while now, and are occasionally getting wrong behaviour ou= t of my system. The symptom I=92m getting is that two servers are concluding that they are = the leader of a particular task/leaderlatch at the same time, braking every= thing in my application. This does not happen too often - but often enough and it is bad enough for = my application. I can get it pretty consistently occurring by restarting on= e of the servers in our 5-server zookeeper ensembles in turns, while having multiple servers queuing up for the same leader latch. My key question is the following: - WHAT should a user of a leaderLatch do when the connectionState goes to s= uspended? My assumption and desired behaviour is that the user should suspend operati= ons - which implies to me that its leadership status is uncertain. (I am ho= lding off all persistent operations for example). But -I think- this also implies that no-one else can become leader yet - we= either have the old-leader still be leader, and no one else, or then the o= ld-leader disappeared and we are in effect leaderless for some time. This will then be followed by a) a reconnect - in which case the old leader can continue its stuff (and o= ptionally double check its leadership status) or b) a lost - in which case the old leader lost its leadership and should rel= ease all its power etc and try again or do something else. Someone else lik= ely became leader in my application by then. The a) or b) is controlled by the SessionTimeout negotiated between the cur= ator/zookeeper client and zookeeper ensemble. Is my thinking correct here? and if so, why is the curator=92s LeaderLatch.handleStateChange(ConnectionS= tate newState) handling both in the same way : setLeadership(false) In my application, a leadership change is a pretty big event, due to the am= ount of work the code does, and I really want leadership to remain between = short connection-breaks - eg. one of the zookeeper servers crashes. Leaders= hip should only be swapped on a sessiontimeout - eg. broken application nod= e, or long network break between the server and the zookeeper servers. I am= thinking to use 90 second as session timeout (so to survive eg. longer GC = breaks and similar without leadership change) - maybe even longer. Is this a bug in leader latch, or should I use something else than leader l= atch, or implement my desired behaviour in a new recipe? kind regards, Robert Kamphuis PS. using zookeeper3.4.5 and curator2.4.0 --_000_CF4F0BAB4F4Bmatthewbrowncitrixcom_ Content-Type: text/html; charset="Windows-1252" Content-ID: <88366D202CE7C4408C812C0687391D69@citrix.com> Content-Transfer-Encoding: quoted-printable
> My assumption and desired behaviour is that the user should suspe= nd operations - which implies to me that its leadership status is uncertain= . (I am holding off all persistent operations for example). 
> But -I think- this also implies that no-one else can become leade= r yet - we either have the old-leader still be leader, and no one else, or = then the old-leader disappeared and we are in effect leaderless for some ti= me.

I think the second part of this is incorrect =96 if client 1 has lost = it's zookeeper connection, it doesn't imply that other clients have also lo= st their zookeeper connection.

So it would be correct for the former leader who now has a suspended c= onnection to cease it's leader activities =96 but other clients who are sti= ll connected to the ensemble may have become the leader due to the suspensi= on of client 1's connection.

If client 1 still acted as if it still might be the leader when it's c= onnection becomes suspended, then you would have two leaders =96 client 1 a= nd whatever client which that still has a healthy ZK connection which grabb= ed the latch.

From the perspective of the zookeeper ensemble, it can't know if your = client is suffering from a "short connection break" or if it has = died altogether =96 so the client's leader role should be treated as lost i= n either case.

From: Robert Kamphuis <Robert.Kamphuis@supercell.com= >
Reply-To: "user@curator.apache.org" <user@curator.apache.org>
Date: Wednesday, March 19, 2014 at = 6:18 AM
To: "user@curator.apache.org" <user@curator.apache.org>
Cc: Robert Kamphuis <Robert.Kamphuis@supercell.com>=
Subject: Confused about the LeaderL= atch - what should happen on ConnectionState.SUSPENDED and ConnectionState.= LOST ?


Hi, 

I have been working on changing our application to work with Zookeep= er and Curator for some while now, and are occasionally getting wrong behav= iour out of my system.
The symptom I=92m getting is that two servers are concluding that they are = the leader of a particular task/leaderlatch at the same time, braking every= thing in my application.
This does not happen too often - but often enough and it is bad enough for = my application. I can get it pretty consistently occurring by restarting on= e of the servers in our 5-server zookeeper ensembles in turns, 
while having multiple servers queuing up for the same leader latch. 
My key question is the following:
- WHAT should a user of a leaderLatch do when the connectionState goes to s= uspended?

My assumption and desired behaviour is that the user should suspend = operations - which implies to me that its leadership status is uncertain.&n= bsp;(I am holding off all persistent operations for example). 
But -I think- this also implies that no-one else can become leader yet - we= either have the old-leader still be leader, and no one else, or then the o= ld-leader disappeared and we are in effect leaderless for some time.
This will then be followed by 
a) a reconnect - in which case the old leader can continue its stuff (and o= ptionally double check its leadership status) or
b) a lost - in which case the old leader lost its leadership and should rel= ease all its power etc and try again or do something else. Someone else lik= ely became leader in my application by then.
The a) or b) is controlled by the SessionTimeout negotiated between the cur= ator/zookeeper client and zookeeper ensemble.

Is my thinking correct here?
and if so, why is the curator=92s LeaderLatch.handleStateChange(ConnectionS= tate newState) handling both in the same way : setLeadership(false)

In my application, a leadership change is a pretty big event, due to= the amount of work the code does, and I really want leadership to remain b= etween short connection-breaks - eg. one of the zookeeper servers crashes. = Leadership should only be swapped on a sessiontimeout - eg. broken application node, or long network break b= etween the server and the zookeeper servers. I am thinking to use 90 second= as session timeout (so to survive eg. longer GC breaks and similar without= leadership change) - maybe even longer.

Is this a bug in leader latch, or should I use something else than l= eader latch, or implement my desired behaviour in a new recipe?

kind regards,
Robert Kamphuis
PS. using zookeeper3.4.5 and curator2.4.0

--_000_CF4F0BAB4F4Bmatthewbrowncitrixcom_--