zookeeper-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ZOOKEEPER-2930) Leader cannot be elected due to network timeout of some members.
Date Fri, 02 Feb 2018 18:01:00 GMT

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-2930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16350736#comment-16350736
] 

ASF GitHub Bot commented on ZOOKEEPER-2930:
-------------------------------------------

GitHub user JonathanO opened a pull request:

    https://github.com/apache/zookeeper/pull/456

    ZOOKEEPER-2930: Leader cannot be elected due to network timeout of some members.

    Move sock.connect() into the async connection worker thread.
    Use initiateConnectionAsync for all connections.
    This prevents connection delays blocking notifications to other nodes.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/transferwise/zookeeper ZOOKEEPER-2930

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/zookeeper/pull/456.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #456
    
----
commit 68e327e112b0910e56b9fa2b3ba948e4179adb6b
Author: Jonathan Oddy <jonathan.oddy@...>
Date:   2018-02-02T15:33:50Z

    ZOOKEEPER-2930: Leader cannot be elected due to network timeout of some members.
    
    Move sock.connect() into the async connection worker thread.
    Use initiateConnectionAsync for all connections.
    This prevents connection delays blocking notifications to other nodes.

----


> Leader cannot be elected due to network timeout of some members.
> ----------------------------------------------------------------
>
>                 Key: ZOOKEEPER-2930
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2930
>             Project: ZooKeeper
>          Issue Type: Bug
>          Components: leaderElection, quorum, server
>    Affects Versions: 3.4.10, 3.5.3, 3.4.11, 3.5.4, 3.4.12
>         Environment: Java 8
> ZooKeeper 3.4.11(from github)
> Centos6.5
>            Reporter: Jiafu Jiang
>            Priority: Critical
>         Attachments: zoo.cfg, zookeeper1.log, zookeeper2.log
>
>
> I deploy a cluster of ZooKeeper with three nodes:
> ofs_zk1:20.10.11.101, 30.10.11.101
> ofs_zk2:20.10.11.102, 30.10.11.102
> ofs_zk3:20.10.11.103, 30.10.11.103
> I shutdown the network interfaces of ofs_zk2 using "ifdown eth0 eth1" command.
> It is supposed that the new Leader should be elected in some seconds, but the fact is,
ofs_zk1 and ofs_zk3 just keep electing again and again, but none of them can become the new
Leader.
> I change the log level to DEBUG (the default is INFO), and restart zookeeper servers
on ofs_zk1 and ofs_zk2 again, but it can not fix the problem.
> I read the log and the ZooKeeper source code, and I think I find the reason.
> When the potential leader(says ofs_zk3) begins the election(FastLeaderElection.lookForLeader()),
it will send notifications to all the servers. 
> When it fails to receive any notification during a timeout, it will resend the notifications,
and double the timeout. This process will repeat until any notification is received or the
timeout reaches a max value.
> The FastLeaderElection.sendNotifications() just put the notification message into a queue
and return. The WorkerSender is responsable to send the notifications.
> The WorkerSender just process the notifications one by one by passing the notifications
to QuorumCnxManager. Here comes the problem, the QuorumCnxManager.toSend() blocks for a long
time when the notification is send to ofs_zk2(whose network is down) and some notifications
(which belongs to ofs_zk1) will thus be blocked for a long time. The repeated notifications
by FastLeaderElection.sendNotifications() just make things worse.
> Here is the related source code:
> {code:java}
>     public void toSend(Long sid, ByteBuffer b) {
>         /*
>          * If sending message to myself, then simply enqueue it (loopback).
>          */
>         if (this.mySid == sid) {
>              b.position(0);
>              addToRecvQueue(new Message(b.duplicate(), sid));
>             /*
>              * Otherwise send to the corresponding thread to send.
>              */
>         } else {
>              /*
>               * Start a new connection if doesn't have one already.
>               */
>              ArrayBlockingQueue<ByteBuffer> bq = new ArrayBlockingQueue<ByteBuffer>(SEND_CAPACITY);
>              ArrayBlockingQueue<ByteBuffer> bqExisting = queueSendMap.putIfAbsent(sid,
bq);
>              if (bqExisting != null) {
>                  addToSendQueue(bqExisting, b);
>              } else {
>                  addToSendQueue(bq, b);
>              }
>              
>              // This may block!!!
>              connectOne(sid);
>                 
>         }
>     }
> {code}
> Therefore, when ofs_zk3 believes that it is the leader, it begins to wait the epoch ack,
but in fact the ofs_zk1 does not receive the notification(which says the leader is ofs_zk3)
because the ofs_zk3 has not sent the notification(which may still exist in the sendqueue of
WorkerSender). At last, the potential leader ofs_zk3 fails to receive the epoch ack in timeout,
so it quits the leader and begins a new election. 
> The log files of ofs_zk1 and ofs_zk3 are attached.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message