From issues-return-550-archive-asf-public=cust-asf.ponee.io@zookeeper.apache.org Fri Aug 2 09:18:02 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id C8709180647 for ; Fri, 2 Aug 2019 11:18:01 +0200 (CEST) Received: (qmail 92979 invoked by uid 500); 2 Aug 2019 09:18:01 -0000 Mailing-List: contact issues-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@zookeeper.apache.org Delivered-To: mailing list issues@zookeeper.apache.org Received: (qmail 92884 invoked by uid 99); 2 Aug 2019 09:18:01 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Aug 2019 09:18:01 +0000 Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 544E0E0220 for ; Fri, 2 Aug 2019 09:18:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 0F45F213F5 for ; Fri, 2 Aug 2019 09:18:00 +0000 (UTC) Date: Fri, 2 Aug 2019 09:18:00 +0000 (UTC) From: "Jan-Philip Gehrcke (JIRA)" To: issues@zookeeper.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (ZOOKEEPER-3466) ZK cluster converges, but does not properly handle client connections (new in 3.5.5) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/ZOOKEEPER-3466?page=3Dcom.atlas= sian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D= 16898723#comment-16898723 ]=20 Jan-Philip Gehrcke commented on ZOOKEEPER-3466: ----------------------------------------------- Hey [~hanm] and others. What can we do to move this forward? Do we need mor= e debug info from here? Thanks for your help! > ZK cluster converges, but does not properly handle client connections (ne= w in 3.5.5) > -------------------------------------------------------------------------= ----------- > > Key: ZOOKEEPER-3466 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3466 > Project: ZooKeeper > Issue Type: Bug > Affects Versions: 3.5.5 > Environment: Linux > Reporter: Jan-Philip Gehrcke > Priority: Major > > Hey, we explore switching from ZooKeeper 3.4.14 to ZooKeeper 3.5.5 in=C2= =A0[https://github.com/dcos/dcos]. > DC/OS coordinates ZooKeeper via Exhibitor. We are not changing anything w= .r.t. Exhibitor for now, and are hoping that we can use ZooKeeper 3.5.5 as = a drop-in replacement for 3.4.14. This seems to work fine=C2=A0when Exhibit= or uses a so-called static ensemble where the individual ZooKeeper instance= s are known a priori. > When Exhibitor however discovers individual ZooKeeper instances ("dynamic= " back-end) then I think we observe a regression where ZooKeeper 3.5.5 can = get into the following bad state (often, but not always): > # three ZooKeeper instances find each other, leader election takes place= =C2=A0(*expected*) > # leader election succeeds: two followers, one leader=C2=A0(*expected*) > # all three ZK instances respond IAMOK to RUOK=C2=A0=C2=A0(*expected*) > # all three ZK instances respond to SRVR (one says "Mode: leader", the o= ther two say "Mode: follower")=C2=A0=C2=A0(*expected*) > # all three ZK instances respond to MNTR and show plausible output (*exp= ected*) > # *{color:#ff0000}Unexpected:{color}* any ZooKeeper client trying to con= nect to any of the three nodes observes a "connection timeout", whereas not= ably this is *not* a TCP connect() timeout. The TCP connect() succeeds, but= then ZK does not seem to send the expected byte sequence to the TCP connec= tion, and the ZK client waits for it via recv() until it hits a timeout con= dition. Examples for two different clients: > ## In Kazoo we specifically hit=C2=A0_Connection time-out: socket time-o= ut during read_ > generated here:=C2=A0[https://github.com/python-zk/kazoo/blob/88b657a097= 7161f3815657878ba48f82a97a3846/kazoo/protocol/connection.py#L249] > ## In zkCli we see=C2=A0=C2=A0_Client session timed out, have not heard = from server in 15003ms for sessionid 0x0, closing socket connection and att= empting reconnect (org.apache.zookeeper.ClientCnxn:main-SendThread(localhos= t:2181))_ > # This state is stable, it will last forever (well, at least for multipl= e hours and we didn't test longer than that). > # In our system the ZooKeeper clients are crash-looping. They retry. Wha= t I have observed is that while they retry the ZK ensemble accumulates outs= tanding requests, here shown from MNTR output (emphasis mine):=C2=A0 > zk_packets_received 2008 > zk_packets_sent 127 > zk_num_alive_connections 18 > zk_outstanding_requests *1880* > # The leader emits log lines confirming session timeout, example: > _[myid:3] INFO [SessionTracker:ZooKeeperServer@398] - Expiring session 0= x2000642b18f0020, timeout of 10000ms exceeded [myid:3] INFO [SessionTracker= :QuorumZooKeeperServer@157] - Submitting global closeSession request for se= ssion 0x2000642b18f0020_ > # In this state, restarting any one of the two ZK followers results in t= he same state (clients don't get data from ZK upon connect). > # In this state, restarting the ZK leader, and therefore triggering a le= ader re-election, almost immediately results in all clients being able to c= onnect to all ZK instances successfully. -- This message was sent by Atlassian JIRA (v7.6.14#76016)