Mailing-List: contact dev-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@zookeeper.apache.org
Date: Sat, 13 May 2017 00:44:04 +0000 (UTC)
From: "Ben Sherman (JIRA)" <jira@apache.org>
To: dev@zookeeper.apache.org
Message-ID: <JIRA.13071710.1494636206000.202317.1494636244021@Atlassian.JIRA>
In-Reply-To: <JIRA.13071710.1494636206000@Atlassian.JIRA>
References: <JIRA.13071710.1494636206000@Atlassian.JIRA> <JIRA.13071710.1494636206527@jira-lw-us.apache.org>
Subject: [jira] [Created] (ZOOKEEPER-2783) follower disconnects and cannot
 reconnect
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Sat, 13 May 2017 00:44:07 -0000

Ben Sherman created ZOOKEEPER-2783:
--------------------------------------

             Summary: follower disconnects and cannot reconnect
                 Key: ZOOKEEPER-2783
                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2783
             Project: ZooKeeper
          Issue Type: Bug
          Components: leaderElection
    Affects Versions: 3.4.10
         Environment: centos 7, AWS EC2
            Reporter: Ben Sherman


We have a 5 node cluster running 3.4.10 we saw this in .8 and .9 as well), and sometimes, a node gets a read timeout, drops all the connections and tries to re-establish itself to the quorum.  It can usually do this in a few seconds, but last night it took almost 15 minutes to reconnect.

These are 5 servers in AWS, and we've tried tuning the timeouts, but the are exceeding any reasonable timeout and still failing.

In the attached logs, 5 is a follower, 3 is the leader.  5 loses connectivity at 11:21:34.  3 sees the disconnect at the same moment.

5 tries to re-establish the quorum, but cannot do it until the connections to the other servers expire at 11:37:02.  After the connections are re-established, 5 connects immediately.

At 11:41:08, the operator restarted the server, and it reconnected normally.

I suspect there is a problem with stale connections to the rest of the quorum - the other services on this box were fine (monitoring, puppet) and able to establish new connections with no problems.

I posed this problem to the zookeeper-users list and was asked to open a ticket.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)