From user-return-12250-archive-asf-public=cust-asf.ponee.io@zookeeper.apache.org Thu Oct 24 22:52:29 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 888C618065D for ; Fri, 25 Oct 2019 00:52:29 +0200 (CEST) Received: (qmail 81244 invoked by uid 500); 24 Oct 2019 22:52:28 -0000 Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@zookeeper.apache.org Delivered-To: mailing list user@zookeeper.apache.org Received: (qmail 81226 invoked by uid 99); 24 Oct 2019 22:52:27 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Oct 2019 22:52:27 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id DB02EC0656 for ; Thu, 24 Oct 2019 22:52:26 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0 X-Spam-Level: X-Spam-Status: No, score=0 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id CXxGPIryOkt2 for ; Thu, 24 Oct 2019 22:52:25 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2607:f8b0:4864:20::b2a; helo=mail-yb1-xb2a.google.com; envelope-from=jerry.hebert@gmail.com; receiver= Received: from mail-yb1-xb2a.google.com (mail-yb1-xb2a.google.com [IPv6:2607:f8b0:4864:20::b2a]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id ED0ED7DDF1 for ; Thu, 24 Oct 2019 22:52:24 +0000 (UTC) Received: by mail-yb1-xb2a.google.com with SMTP id h202so175544ybg.13 for ; Thu, 24 Oct 2019 15:52:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=iGqiRTiKKYWAkPQvYhDKm2jX/uLADZssz8SCmSAF0CU=; b=CNQjTr7KDNPmiOW3TOmEt9br3eBZqzRmn+/ll7VtcAhbY9mHHyFoqj6gWXhB7wrGew EaW6mI3MW8B+anw/mHtpvPC0hm/JU2Yha4S4BaACf3u9Mumz2AEac2sx2RDDO2SGYnlF +UvuCPbGy6OtaUhiHS0W6wsd0q3TfR5aOWpeKrQAZ4xs06Eq9NQMar/cIu5b+wghXqh2 jMxe495kriSVQf8SrbG8tA0ATe1ELTIhanOS0CXD3V6BB/OmZD4EgE/oIUMFq3I957Xv H6Y2vlGyRxyMDJyqBxCnHkbEmc21AdNdES6CjtwN1nMhNDoNJLG8xYOTRMFHpd7cHejG M68g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=iGqiRTiKKYWAkPQvYhDKm2jX/uLADZssz8SCmSAF0CU=; b=TCuQd/c0rO1Luy/8ti/xZ6UXM9K4KrlsrZn+PzO0hwdCUjUaHOnc5B2Iy/DLT7Jubu hZy5mTM8+jCdOoqV0f6GYmw/SJbhVdxsrIzCzbc9s3Eim93m0f7CiMtHsMXfzzv9uMlQ +EbAxsmkOGXVU4dN/dlHHpdo+OmE2Y/JKEyTjXNrH9viZh7mQ+HqNt8U7ZHXkColvF3F rzAwnr59tTBK3DKV2MC7erlo7/W2BHDm+P4F5DzXpJZ7Gu3LJNpYkY9M03EwEvTRc7Mv X0ED0NZTAKsyiED0KuPqYPYVIlTrPmGMwVECydaMxLUagQ424oRc3sokM3j80nChJ8HD L34w== X-Gm-Message-State: APjAAAXvwGtltiUgqb6yvV+Btirc6/KBzOkzQU64LhPhEDs1ZwptWVAL ouTanZUJ6pPlTkXHltE1vMjDqWVDGKavtHFTHtYDHavA X-Google-Smtp-Source: APXvYqzlNtALlbTshpOAZIqMvmJutHtY85kcFNUgYol96mPt7yzeuF6/d0/n6HxzSKpw8PTc+XU/nbeD56OChQ3PTV8= X-Received: by 2002:a25:d604:: with SMTP id n4mr696240ybg.472.1571957537894; Thu, 24 Oct 2019 15:52:17 -0700 (PDT) MIME-Version: 1.0 From: Jerry Hebert Date: Thu, 24 Oct 2019 15:52:01 -0700 Message-ID: Subject: "Connections" incorrectly reported as 1 (3.5.5) To: user@zookeeper.apache.org Content-Type: multipart/alternative; boundary="000000000000a5f7800595afe209" --000000000000a5f7800595afe209 Content-Type: text/plain; charset="UTF-8" Hey all, I've upgraded one of my ensembles to 3.5.5 now (3.5.5-390fe37ea45dee01bf87dc1c042b5e3dcce88653, specifically). All of my metrics appeared to be healthy but after the migration, I noticed that the new ensemble has *all 5 nodes reporting a connection count of 1* (via the "stat" 4ltr command as well as zk_num_alive_connections from the "mntr" output). The servers are clearly receiving traffic: I can see node counts going up and down and I can see clients making changes to various keys. I can also monitor netstat for 2181 connections and again see connections fluctuating per usual but I still see "Connections: 1" in stats. This translates into our Datadog agent reporting connections as 1 too. I've been reading through the code to try to understand how this may be possible but it's a bit of a slog as I'm unfamiliar with it and I've found myself digging into Netty now. I pasted a couple of possibly relevant log lines below. In particular, note that "secure" is false here and I noticed that the conx count is split in the code depending on whether or not you're in secure mode. I also find it odd that I'm seeing 0:0:0:0:0:0:0:0 in the logs which looks like ipv6 to me and I'm using ipv4 (or at least, I partially am...). I also don't understand the zxid expectation mismatch. 2019-10-24 22:12:08,411 [myid:9] - WARN [QuorumPeer[myid=9](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Follower@125] - Got zxid 0x2f00000001 expected 0x1 2019-10-24 22:11:57,349 [myid:9] - INFO [main:ServerCnxnFactory@135] - Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory Any advice would be greatly appreciated. I don't feel comfortable leaving this server as-is given that it's misreporting connections. Something is definitely wrong. Thanks in advance! Jerry --000000000000a5f7800595afe209--