curator-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From chao chu <chuchao...@gmail.com>
Subject Re: Leader Latch recovery after suspended state
Date Mon, 10 Mar 2014 15:41:04 GMT
Hi JZ,

Sorry for being dense here, but my point is that: Suppose the leader latch
does NOT setLeadership(false) on receiving SUSPENDED, then what you
mentioned below won't happen, right?

*>>> Further, if zk2/zk3 are communicating then they are in Quorum.
Therefore, latch2 will declare that it is leader*

this way we can avoid the unnecessary leader switch due to the transient
latch1 <--> zk1 connection loss (i.e., the reboot of zk1 somehome) if the
latch1 can re-connect to the zk ensemble in time.


On Mon, Mar 10, 2014 at 11:28 PM, Jordan Zimmerman <
jordan@jordanzimmerman.com> wrote:

> In step 3, during the time that zk1 is partitioned latch1 cannot know if
> it's leader or not. Further, if zk2/zk3 are communicating then they are in
> Quorum. Therefore, latch2 will declare that it is leader. So, latch1 MUST
> go into a non-leader state. It doesn't matter if latch1 reconnects before
> the session expires. The issue is how it manages its state while it is in
> partition. To your point, latch1 could check to see if it's still leader
> when it reconnects but I'm not sure that there's a huge win here. Any
> clients of the latch will have to have handled the leadership loss during
> the partition.
>
> -JZ
>
>
> From: chao chu chuchao333@gmail.com
> Reply: user@curator.apache.org user@curator.apache.org
> Date: March 10, 2014 at 10:21:29 AM
> To: user user@curator.apache.org
>
> Subject:  Re: Leader Latch recovery after suspended state
>
>  Thanks for your reply first.
>
>  >>> Technically, this is already the case. When a RECONNECTED is
> received, LeaderLatch will attempt to regain leadership. The problem is
> that when there is a network partition there is no way to guarantee that
> you are still the leader. If there is Quorum in another segment of the
> cluster a new leader might be elected there.
>
>  I don't quite understand what you meant here. But let me explain the
> case I mentioned in detail:
>
> 1. let's say there is a 3-server zk ensemble {zk1, zk2, zk3}, and two
> participants for the leader electoin {leader latch1, leader latch2}
> 2. latch1 is the current leader, and connected to zk1
> 3. the connection latch1 <--> zk1 broken somehow
> 4. but sooner (within the session timeout), latch1 re-connected (maybe to
> zk2)
>
> I guess the problem is that in LeaderLatch, it setLeadership(false)
> (meaning that there must be leader changes) as long as a SUSPEND state (or
> actually a ZK DISCONNECTED event).
>
> while in this case, ideally (just my personal thinking), since latch1
> re-connected in time, its znode will still be there, no others (latch2 in
> this example) will observe any events due to this, and once re-connected,
> if it detected that its znode still there, checkLeadership should still set
> it as leader, and no leader changes (no 'isLeader' or 'notLeader' will be
> called) during this whole process. Thus, we can avoid the unnecessary
> leader switch (which, as I mentioned, can be very expensive in most of
> cases).
>
> does this make any sense to you? thanks
>
>
>
> On Mon, Mar 10, 2014 at 10:52 PM, Jordan Zimmerman <
> jordan@jordanzimmerman.com> wrote:
>
>>  Please provide an implementation/fix and submit a pull request on
>> Github.
>>
>>   I also have a related question about not only re-use the znode, but
>> imho, It would be great that LeaderLatch can survive from teomprary
>> ConnectionLossException (i.e., due to transient network issue).
>>
>>  Technically, this is already the case. When a RECONNECTED is received,
>> LeaderLatch will attempt to regain leadership. The problem is that when
>> there is a network partition there is no way to guarantee that you are
>> still the leader. If there is Quorum in another segment of the cluster a
>> new leader might be elected there.
>>
>> -JZ
>>
>> From: chao chu chuchao333@gmail.com
>> Reply: user@curator.apache.org user@curator.apache.org
>> Date: March 10, 2014 at 9:39:50 AM
>> To: user@curator.incubator.apache.org user@curator.incubator.apache.org
>> Subject:  Re: Leader Latch recovery after suspended state
>>
>>   Hi,
>>
>> Just want to see if there is any progress on this?
>>
>> I also have a related question about not only re-use the znode, but imho,
>> It would be great that LeaderLatch can survive from teomprary
>> ConnectionLossException (i.e., due to transient network issue).
>>
>> I guess in most cases, the context switch due to leader re-election is
>> quite expensive, we might not want to do that just because of some
>> transient issue. if the current leader can re-connect within the session
>> timeout, it should still hold the leadership and no leader change would
>> happen during between. The similar rational like the differences between
>> ConnestionLossException (which is recoverable) and SessionExipredException
>> (which is not recoverable).
>>
>> what are your thoughts on this? Thanks a lot!
>>
>> Regards,
>>
>>
>> On Wed, Aug 21, 2013 at 2:05 AM, Jordan Zimmerman <
>> jordan@jordanzimmerman.com> wrote:
>>
>>>  Yes, I was suggesting how to patch Curator.
>>>
>>>  On Aug 20, 2013, at 10:59 AM, Calvin Jia <jia.calvin@gmail.com> wrote:
>>>
>>>  Currently this is not supported in the Curator library, but the
>>> Curator library (specifically leader latch's reset method) is the
>>> correct/logical place to add this feature if I want it?
>>>
>>>
>>> On Tue, Aug 20, 2013 at 10:34 AM, Jordan Zimmerman <
>>> jordan@jordanzimmerman.com> wrote:
>>>
>>>>  On reset() it could check to see if its node still exists. It would
>>>> make the code a lot more complicated though.
>>>>
>>>> -JZ
>>>>
>>>>  On Aug 20, 2013, at 10:25 AM, Calvin Jia <jia.calvin@gmail.com> wrote:
>>>>
>>>>  A leader latch enters the suspended state after failing to receive a
>>>> response from the first ZK machine it heartbeats to (takes 2 thirds of the
>>>> timeout). For the last 1 third, it tries to contact another ZK machine. If
>>>> it is successful, it will enter the state reconnected.
>>>>
>>>> However, on reconnect, despite the fact the original node it created in
>>>> ZK is still there, it will create another ephemeral-sequential node (the
>>>> reset method is called). This means it will relinquish leadership, if there
>>>> is another machine with a latch in the same path.
>>>>
>>>> Is there any way to reconnect and reuse the original ZK node?
>>>>
>>>>  Thanks!
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>> --
>> ChuChao
>>
>>
>
>
> --
> ChuChao
>
>


-- 
ChuChao

Mime
View raw message