Return-Path: X-Original-To: apmail-curator-user-archive@minotaur.apache.org Delivered-To: apmail-curator-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E2CC5C9A1 for ; Mon, 10 Mar 2014 15:41:55 +0000 (UTC) Received: (qmail 80928 invoked by uid 500); 10 Mar 2014 15:41:55 -0000 Delivered-To: apmail-curator-user-archive@curator.apache.org Received: (qmail 80408 invoked by uid 500); 10 Mar 2014 15:41:52 -0000 Mailing-List: contact user-help@curator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@curator.apache.org Delivered-To: mailing list user@curator.apache.org Received: (qmail 80123 invoked by uid 99); 10 Mar 2014 15:41:49 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Mar 2014 15:41:49 +0000 X-ASF-Spam-Status: No, hits=1.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of chuchao333@gmail.com designates 209.85.220.175 as permitted sender) Received: from [209.85.220.175] (HELO mail-vc0-f175.google.com) (209.85.220.175) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Mar 2014 15:41:45 +0000 Received: by mail-vc0-f175.google.com with SMTP id lh14so5105352vcb.20 for ; Mon, 10 Mar 2014 08:41:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=R5RPnZxlItfGKWUGb2iYKVWF4jj+dAfdJCiFaBO0kYc=; b=iWQf8cgC3vuYKha2Tf7LbsIOMBWhL7XXoIY69t5lWS2MXIlSkFHq4EAS7z7yE0bxYj 8739rs672Ghm1VxgikUppOyBEYDCKWw7o4Be5u2vdHqiIUYQz2tG0XLdGDPnq3We20ny E/UgyfUMjT2CcVqv7gwhlrP/DReO970p8EwD2C0JUo9TbgSe717dJGaQzD06And/psq0 XRnBDchKksce0JVJNQ5r7W/zm7iuEdJvLaG7Q78IsX7siY6dCeWlsaMJb9D4EY6MLvw5 DBFnSVsj/ia/RwvUQHH7kUYgjpFfySYukYbSSc4Rjn7xqPJp+MHw5InUS27nSlXiE9pE BZKA== X-Received: by 10.52.126.107 with SMTP id mx11mr631437vdb.41.1394466084730; Mon, 10 Mar 2014 08:41:24 -0700 (PDT) MIME-Version: 1.0 Received: by 10.220.167.197 with HTTP; Mon, 10 Mar 2014 08:41:04 -0700 (PDT) In-Reply-To: References: <308E4CC1-6514-4CCC-BE0E-C3ABD0A7C452@jordanzimmerman.com> <1D572118-6678-42F9-A63E-32F16D025007@jordanzimmerman.com> From: chao chu Date: Mon, 10 Mar 2014 23:41:04 +0800 Message-ID: Subject: Re: Leader Latch recovery after suspended state To: user@curator.apache.org Content-Type: multipart/alternative; boundary=bcaec520f689a1be2204f4426f01 X-Virus-Checked: Checked by ClamAV on apache.org --bcaec520f689a1be2204f4426f01 Content-Type: text/plain; charset=ISO-8859-1 Hi JZ, Sorry for being dense here, but my point is that: Suppose the leader latch does NOT setLeadership(false) on receiving SUSPENDED, then what you mentioned below won't happen, right? *>>> Further, if zk2/zk3 are communicating then they are in Quorum. Therefore, latch2 will declare that it is leader* this way we can avoid the unnecessary leader switch due to the transient latch1 <--> zk1 connection loss (i.e., the reboot of zk1 somehome) if the latch1 can re-connect to the zk ensemble in time. On Mon, Mar 10, 2014 at 11:28 PM, Jordan Zimmerman < jordan@jordanzimmerman.com> wrote: > In step 3, during the time that zk1 is partitioned latch1 cannot know if > it's leader or not. Further, if zk2/zk3 are communicating then they are in > Quorum. Therefore, latch2 will declare that it is leader. So, latch1 MUST > go into a non-leader state. It doesn't matter if latch1 reconnects before > the session expires. The issue is how it manages its state while it is in > partition. To your point, latch1 could check to see if it's still leader > when it reconnects but I'm not sure that there's a huge win here. Any > clients of the latch will have to have handled the leadership loss during > the partition. > > -JZ > > > From: chao chu chuchao333@gmail.com > Reply: user@curator.apache.org user@curator.apache.org > Date: March 10, 2014 at 10:21:29 AM > To: user user@curator.apache.org > > Subject: Re: Leader Latch recovery after suspended state > > Thanks for your reply first. > > >>> Technically, this is already the case. When a RECONNECTED is > received, LeaderLatch will attempt to regain leadership. The problem is > that when there is a network partition there is no way to guarantee that > you are still the leader. If there is Quorum in another segment of the > cluster a new leader might be elected there. > > I don't quite understand what you meant here. But let me explain the > case I mentioned in detail: > > 1. let's say there is a 3-server zk ensemble {zk1, zk2, zk3}, and two > participants for the leader electoin {leader latch1, leader latch2} > 2. latch1 is the current leader, and connected to zk1 > 3. the connection latch1 <--> zk1 broken somehow > 4. but sooner (within the session timeout), latch1 re-connected (maybe to > zk2) > > I guess the problem is that in LeaderLatch, it setLeadership(false) > (meaning that there must be leader changes) as long as a SUSPEND state (or > actually a ZK DISCONNECTED event). > > while in this case, ideally (just my personal thinking), since latch1 > re-connected in time, its znode will still be there, no others (latch2 in > this example) will observe any events due to this, and once re-connected, > if it detected that its znode still there, checkLeadership should still set > it as leader, and no leader changes (no 'isLeader' or 'notLeader' will be > called) during this whole process. Thus, we can avoid the unnecessary > leader switch (which, as I mentioned, can be very expensive in most of > cases). > > does this make any sense to you? thanks > > > > On Mon, Mar 10, 2014 at 10:52 PM, Jordan Zimmerman < > jordan@jordanzimmerman.com> wrote: > >> Please provide an implementation/fix and submit a pull request on >> Github. >> >> I also have a related question about not only re-use the znode, but >> imho, It would be great that LeaderLatch can survive from teomprary >> ConnectionLossException (i.e., due to transient network issue). >> >> Technically, this is already the case. When a RECONNECTED is received, >> LeaderLatch will attempt to regain leadership. The problem is that when >> there is a network partition there is no way to guarantee that you are >> still the leader. If there is Quorum in another segment of the cluster a >> new leader might be elected there. >> >> -JZ >> >> From: chao chu chuchao333@gmail.com >> Reply: user@curator.apache.org user@curator.apache.org >> Date: March 10, 2014 at 9:39:50 AM >> To: user@curator.incubator.apache.org user@curator.incubator.apache.org >> Subject: Re: Leader Latch recovery after suspended state >> >> Hi, >> >> Just want to see if there is any progress on this? >> >> I also have a related question about not only re-use the znode, but imho, >> It would be great that LeaderLatch can survive from teomprary >> ConnectionLossException (i.e., due to transient network issue). >> >> I guess in most cases, the context switch due to leader re-election is >> quite expensive, we might not want to do that just because of some >> transient issue. if the current leader can re-connect within the session >> timeout, it should still hold the leadership and no leader change would >> happen during between. The similar rational like the differences between >> ConnestionLossException (which is recoverable) and SessionExipredException >> (which is not recoverable). >> >> what are your thoughts on this? Thanks a lot! >> >> Regards, >> >> >> On Wed, Aug 21, 2013 at 2:05 AM, Jordan Zimmerman < >> jordan@jordanzimmerman.com> wrote: >> >>> Yes, I was suggesting how to patch Curator. >>> >>> On Aug 20, 2013, at 10:59 AM, Calvin Jia wrote: >>> >>> Currently this is not supported in the Curator library, but the >>> Curator library (specifically leader latch's reset method) is the >>> correct/logical place to add this feature if I want it? >>> >>> >>> On Tue, Aug 20, 2013 at 10:34 AM, Jordan Zimmerman < >>> jordan@jordanzimmerman.com> wrote: >>> >>>> On reset() it could check to see if its node still exists. It would >>>> make the code a lot more complicated though. >>>> >>>> -JZ >>>> >>>> On Aug 20, 2013, at 10:25 AM, Calvin Jia wrote: >>>> >>>> A leader latch enters the suspended state after failing to receive a >>>> response from the first ZK machine it heartbeats to (takes 2 thirds of the >>>> timeout). For the last 1 third, it tries to contact another ZK machine. If >>>> it is successful, it will enter the state reconnected. >>>> >>>> However, on reconnect, despite the fact the original node it created in >>>> ZK is still there, it will create another ephemeral-sequential node (the >>>> reset method is called). This means it will relinquish leadership, if there >>>> is another machine with a latch in the same path. >>>> >>>> Is there any way to reconnect and reuse the original ZK node? >>>> >>>> Thanks! >>>> >>>> >>>> >>> >>> >> >> >> -- >> ChuChao >> >> > > > -- > ChuChao > > -- ChuChao --bcaec520f689a1be2204f4426f01 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi JZ,

Sorry for being dense here, but = my point is that: Suppose the leader latch does NOT setLeadership(false) on= receiving SUSPENDED, then what you mentioned below won't happen, right= ?

>>> Further, if zk2/zk3 are communicating = then they are in Quorum. Therefore, latch2 will declare that it is leader

this way we can = avoid the unnecessary leader switch due to the transient latch1 <--> = zk1 connection loss (i.e., the reboot of zk1 somehome) if the latch1 can re= -connect to the zk ensemble in time.


On Mon= , Mar 10, 2014 at 11:28 PM, Jordan Zimmerman <jordan@jordanzimme= rman.com> wrote:
In step 3, during the time that zk1 is partitioned latch1 cannot know if it= ’s leader or not. Further, if zk2/zk3 are communicating then they are= in Quorum. Therefore, latch2 will declare that it is leader. So, latch1 MU= ST go into a non-leader state. It doesn’t matter if latch1 reconnects= before the session expires. The issue is how it manages its state while it= is in partition. To your point, latch1 could check to see if it’s st= ill leader when it reconnects but I’m not sure that there’s a h= uge win here. Any clients of the latch will have to have handled the leader= ship loss during the partition.

-JZ


From: chao= chu chuch= ao333@gmail.com
Reply: user@curator.apache.org user@curator.apache.o= rg
Date: March 10, 2014 at 10:21:29 AM
To:&nb= sp;user user@curator.apache.org

Subject:=   Re: Leader Latch recovery after suspended state

Thanks for your reply first.

>>> Technically, this is already the case. When a RECONNECTED is received, LeaderLatch will attempt to regain leadership. The problem is that when there is a network partition there is no way to guarantee that you are still the leader. If there is Quorum in another segment of the cluster a new leader might be elected there.

I don't quite understand what you meant here. But let me explain the case I mentioned in detail:

=
1. let's say there is a 3-server zk ensemble {zk1, zk2, zk3}, and two participants for the leader electoin {leader latch1, leader latch2}
2. latch1 is the current leader, and connected to zk1
3. the connection latch1 <--> zk1 broken somehow
4. but sooner (within the session timeout), latch1 re-connected (maybe to zk2)

=
I guess the problem is that in LeaderLatch, it setLeadership(false) (meaning that there must be leader changes) as long as a SUSPEND state (or actually a ZK DISCONNECTED event).

=
while in this case, ideally (just my personal thinking), since latch1 re-connected in time, its znode will still be there, no others (latch2 in this example) will observe any events due to this, and once re-connected, if it detected that its znode still there, checkLeadership should still set it as leader, and no leader changes (no 'isLeader' or 'notLeader' will be called) durin= g this whole process. Thus, we can avoid the unnecessary leader switch (which, as I mentioned, can be very expensive in most of cases).

=
does this make any sense to you? thanks



On Mon, Mar 10, 2014 at 10:52 PM, Jordan Zimmerman <jordan@jordanzimmerman.com> wrote:
Please provide an implementation/fix and submit a pull request on Github.

I also have a related question about not only re-use the znode, but imho, It would be great that LeaderLatch can survive from teomprary ConnectionLossException (i.e., due to transient network issue). 
Technically, this is already the case. When a RECONNECTED is received, LeaderLatch will attempt to regain leadership. The problem is that when there is a network partition there is no way to guarantee that you are still the leader. If there is Quorum in another segment of the cluster a new leader might be elected there.

-JZ

From: chao chu chuchao333@gmail.com
Reply: user@curator.apache.org user@curator.apache.org
Date: March 10, 2014 at 9:39:50 AM
To: user@curator.incubator.apache.org user@curator.incubator= .apache.org
Subject:  Re: Leader Latch recovery after suspended state

Hi,

Just want to see if there is any progress on this?

I also have a related question about not only re-use the znode, but imho, It would be great that LeaderLatch can survive from teomprary ConnectionLossException (i.e., due to transient network issue). 

I guess in most cases, the context switch due to leader re-election is quite expensive, we might not want to do that just because of some transient issue. if the current leader can re-connect within the session timeout, it should still hold the leadership and no leader change would happen during between. The similar rational like the differences between ConnestionLossException (which is recoverable) and SessionExipredException (which is not recoverable).

what are your thoughts on this? Thanks a lot!

Regards,
On Wed, Aug 21, 2013 at 2:05 AM, Jordan Zimmerman <jordan@jordanzimmerman.com> wrote:
Yes, I was suggesting how to patch Curator.

On Aug 20, 2013, at 10:59 AM, Calvin Jia <jia.calvin@gmail.com> wrote:

Currently this is not supported in the Curator library, but the Curator library (specifically leader latch's reset method) is the correct/logical place to add this feature if I want it?


On Tue, Aug 20, 2013 at 10:34 AM, Jordan Zimmerman <jordan@jordanzimmerman.com> wrote:
On reset() it could check to see if its node still exists. It would make the code a lot more complicated though.

-JZ

On Aug 20, 2013, at 10:25 AM, Calvin Jia <jia.calvin@gmail.com> wrote:

A leader latch enters the suspended state after failing to receive a response from the first ZK machine it heartbeats to (takes 2 thirds of the timeout). For the last 1 third, it tries to contact another ZK machine. If it is successful, it will enter the state reconnected.

However, on reconnect, despite the fact the original node it created in ZK is still there, it will create another ephemeral-sequential node (the reset method is called). This means it will relinquish leadership, if there is another machine with a latch in the same path.

Is there any way to reconnect and reuse the original ZK node?

Thanks!






--
ChuChao



--
ChuChao


--
ChuChao
--bcaec520f689a1be2204f4426f01--