hadoop-zookeeper-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vishal K (JIRA)" <j...@apache.org>
Subject [jira] Commented: (ZOOKEEPER-335) zookeeper servers should commit the new leader txn to their logs.
Date Tue, 22 Jun 2010 02:47:55 GMT

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881028#action_12881028
] 

Vishal K commented on ZOOKEEPER-335:
------------------------------------

Hi,

I enabled tracing and did some more debugging. Looks like the restarted peer (and trying to
join the cluster) determines that it is a leader and increments its epoch. However, rest of
the nodes don't acknowledge this node as the leader, and hence, have an older epoch. I will
attache the log. Unfortunately, I don't have traces from other nodes. I will repeat the experiment
later and attache logs from other nodes. 

Scenario:
- Form a 3 node cluster. This is not just ZK cluster. It also involves our application cluster
that uses ZK.
- Kill one of the follower
- After a minute or so restart follower
- Follower rejects leader with "Leader epoch y is less than our epoch y + 1"

>From logs:

a) Peer X restarts and starts leader election.
a) For a small window of time, X thinks that it is the new leader! During this window, for
some reason, rest of the nodes tell X that they are also trying to find a leader. I.e., all
3 nodes are in LOOKING state. After seeing that all 3 nodes are in LOOKING state, X decides
to be a leader?

   155 2010-06-20 23:22:46,421 - DEBUG [WorkerSender Thread:QuorumCnxManager@346] - Opening
channel to server 1
   156 2010-06-20 23:22:46,423 - DEBUG [WorkerReceiver Thread:FastLeaderElection$Messenger$WorkerReceiver@214]
- Receive new notification message. My id = 0
   157 2010-06-20 23:22:46,424 - INFO  [QuorumPeer:/0.0.0.0:2181:FastLeaderElection@689] -
Notification: 0, 77309411393, 1, 0, LOOKING, LOOKING, 0
   158 2010-06-20 23:22:46,424 - DEBUG [QuorumPeer:/0.0.0.0:2181:FastLeaderElection@495] -
id: 0, proposed id: 0, zxid: 77309411393, proposed zxid: 77309411393
   159 2010-06-20 23:22:46,424 - DEBUG [QuorumPeer:/0.0.0.0:2181:FastLeaderElection@717] -
Adding vote: From = 0, Proposed leader = 0, Porposed zxid = 77309411393, Proposed epoch =
1
   160 2010-06-20 23:22:46,426 - INFO  [WorkerSender Thread:QuorumCnxManager@162] - Have smaller
server identifier, so dropping the connection: (1, 0)
   161 2010-06-20 23:22:46,426 - DEBUG [WorkerSender Thread:QuorumCnxManager@346] - Opening
channel to server 2
   162 2010-06-20 23:22:46,427 - DEBUG [Thread-1:QuorumCnxManager$Listener@445] - Connection
request /192.168.1.182:46701
   163 2010-06-20 23:22:46,427 - DEBUG [Thread-1:QuorumCnxManager$Listener@448] - Connection
request: 0
   164 2010-06-20 23:22:46,428 - DEBUG [Thread-1:QuorumCnxManager$SendWorker@504] - Address
of remote peer: 1
   165 2010-06-20 23:22:46,428 - INFO  [WorkerSender Thread:QuorumCnxManager@162] - Have smaller
server identifier, so dropping the connection: (2, 0)
   166 2010-06-20 23:22:46,431 - DEBUG [WorkerReceiver Thread:FastLeaderElection$Messenger$WorkerReceiver@214]
- Receive new notification message. My id = 0
   167 2010-06-20 23:22:46,432 - INFO  [QuorumPeer:/0.0.0.0:2181:FastLeaderElection@689] -
Notification: 1, 77309411372, 1, 0, LOOKING, LOOKING, 1
   168 2010-06-20 23:22:46,432 - DEBUG [QuorumPeer:/0.0.0.0:2181:FastLeaderElection@495] -
id: 1, proposed id: 0, zxid: 77309411372, proposed zxid: 77309411393
   169 2010-06-20 23:22:46,432 - DEBUG [QuorumPeer:/0.0.0.0:2181:FastLeaderElection@717] -
Adding vote: From = 1, Proposed leader = 1, Porposed zxid = 77309411372, Proposed epoch =
1
   170 2010-06-20 23:22:46,436 - DEBUG [Thread-1:QuorumCnxManager$Listener@445] - Connection
request /192.168.1.183:44310
   171 2010-06-20 23:22:46,436 - DEBUG [Thread-1:QuorumCnxManager$Listener@448] - Connection
request: 0
   172 2010-06-20 23:22:46,436 - DEBUG [Thread-1:QuorumCnxManager$SendWorker@504] - Address
of remote peer: 2
   173 2010-06-20 23:22:46,440 - DEBUG [WorkerReceiver Thread:FastLeaderElection$Messenger$WorkerReceiver@214]
- Receive new notification message. My id = 0
   174 2010-06-20 23:22:46,440 - INFO  [QuorumPeer:/0.0.0.0:2181:FastLeaderElection@689] -
Notification: 2, 73014444097, 1, 0, LOOKING, LOOKING, 2
   175 2010-06-20 23:22:46,440 - DEBUG [QuorumPeer:/0.0.0.0:2181:FastLeaderElection@495] -
id: 2, proposed id: 0, zxid: 73014444097, proposed zxid: 77309411393
   176 2010-06-20 23:22:46,441 - DEBUG [QuorumPeer:/0.0.0.0:2181:FastLeaderElection@717] -
Adding vote: From = 2, Proposed leader = 2, Porposed zxid = 73014444097, Proposed epoch =
1
   177 2010-06-20 23:22:46,441 - INFO  [QuorumPeer:/0.0.0.0:2181:QuorumPeer@647] - LEADING

b) As a result X  increments its epoch. Worse, since this node decided to be a leader, it
starts doing transactions. The first set of transactions start removing all ephemeral nodes.
But these transactions are only done locally. Other peers do not ack these transactions since
they know that this peer is not the leader.

c) After a few seconds (8 secs), X relinquishes leadership since it does not receive any ack
from rest of the peers
d) It starts leader election again. This time, rests of the peers respond correctly informing
X that a leader already exists.
e) X determines that it should now follow the leader and starts handshake with the leader.
However, X has incremented its epoch in the mean time. X thinks that the epoch sent by the
leader during handshake is older than the epoch it has noticed before. X aborts connection
with the leader.


> zookeeper servers should commit the new leader txn to their logs.
> -----------------------------------------------------------------
>
>                 Key: ZOOKEEPER-335
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-335
>             Project: Zookeeper
>          Issue Type: Bug
>          Components: server
>    Affects Versions: 3.1.0
>            Reporter: Mahadev konar
>            Assignee: Mahadev konar
>            Priority: Blocker
>             Fix For: 3.4.0
>
>         Attachments: zk.log.gz, zklogs.tar.gz
>
>
> currently the zookeeper followers do not commit the new leader election. This will cause
problems in a failure scenarios with a follower acking to the same leader txn id twice, which
might be two different intermittent leaders and allowing them to propose two different txn's
of the same zxid.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message