From user-return-12251-archive-asf-public=cust-asf.ponee.io@zookeeper.apache.org Thu Oct 24 23:20:29 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 14F2218065D for ; Fri, 25 Oct 2019 01:20:28 +0200 (CEST) Received: (qmail 23470 invoked by uid 500); 24 Oct 2019 23:20:27 -0000 Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@zookeeper.apache.org Delivered-To: mailing list user@zookeeper.apache.org Received: (qmail 23398 invoked by uid 99); 24 Oct 2019 23:20:27 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Oct 2019 23:20:27 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 88F98C02C7 for ; Thu, 24 Oct 2019 23:20:26 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.002 X-Spam-Level: X-Spam-Status: No, score=0.002 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id P6dOa9MpAgut for ; Thu, 24 Oct 2019 23:20:25 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.161.50; helo=mail-yw1-f50.google.com; envelope-from=jerry.hebert@gmail.com; receiver= Received: from mail-yw1-f50.google.com (mail-yw1-f50.google.com [209.85.161.50]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id 51E65C1980 for ; Thu, 24 Oct 2019 23:20:25 +0000 (UTC) Received: by mail-yw1-f50.google.com with SMTP id k127so169861ywc.6 for ; Thu, 24 Oct 2019 16:20:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=XsdYB/PH4rAQFJZ3Esn+nX1bJbXJaOyo0oW20r3vrlM=; b=rjEN+3/j6nP8KD+KnBQcBMVV/35kMMx10ZmjdEmT8/vlFwOHy4vRBl+aJGFt5Zkrvo 4OCz3pf8BeETMO14QTuDEHkuf26NR4zKvugg4+wQe9pQT/pA+bN7V4bgqyml7smNPsqQ ThXD1OsprmNnPxgP3AKKXCk+kLQyq9awLV0eBiyQRbUK/810LqGJbaf3nNjUm6Tl7y2Z frruKdfvVh7gUw20yEHopFwlfmAgGh0qc7qgGGLgJzL7w0GJZvlCxgdFZ+t7CuqxQ+cj lzkw0VvrWT5+abWW/cIoj5mAT0GikEFjyk2uc2nIBUcq3eC/G7RB1sVvwzCPA9xTxCvu muMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=XsdYB/PH4rAQFJZ3Esn+nX1bJbXJaOyo0oW20r3vrlM=; b=QX15ortHY/AnbLIOK0RDdyfJtYlJ5/cS7ktT+FyK6acR6jDmJwOu2v3y1tMfu2Oy2U ctb/yVLP//BY7C6+4tIRSddt73WO/icCl+jQx6WpdCOgP2urPZBpVFlmMnvVd9/aYBTS JHd4rDWPfbRyCH5fOlS0hAhmxJVMnUzTbKBwVjpRESso8ADiOamEMcrIj0vcintgKTgn JC/1R2ZM0eMmg2nL42ugMs9stRtZB4S9cMSP+5Nk8BOE/i/VDXWIiKwxoHJgMI3YPQLT rGZw36WqYJatFEmW4uJGkf2eIbfp0VjPp/l1Uf6sQO6DHgD3RqDPe2hVfsg0nEaux+xw TzpA== X-Gm-Message-State: APjAAAWUitOBLQJyDi/MPvkIp9E+N/f9078XczGuT+hLY1nZNH5YOcs/ 9r4N71fZQkr2gUUfCjIUN1kzF1Se+6/OEwobYIdITA== X-Google-Smtp-Source: APXvYqyIbiKXrTCfdqB5TTgFLfFeYwQE/fOo0wYx3a3BmZvtPSo8aSVcnQQuiPKlCHfUMFKqwKSgK76O4DJMJH8GstY= X-Received: by 2002:a81:5ed4:: with SMTP id s203mr15255ywb.485.1571959218773; Thu, 24 Oct 2019 16:20:18 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Jerry Hebert Date: Thu, 24 Oct 2019 16:20:02 -0700 Message-ID: Subject: Re: "Connections" incorrectly reported as 1 (3.5.5) To: user@zookeeper.apache.org Content-Type: multipart/alternative; boundary="000000000000d628120595b04695" --000000000000d628120595b04695 Content-Type: text/plain; charset="UTF-8" blah, user error (of course?). I don't exactly understand how this was working but all of my new servers were receiving requests because I had one of the old ensemble's servers still running and even though it had been removed (via `reconfig`), it was still accepting connections and it seems like it maybe was forwarding those requests to the new ensemble or maybe all of those 2181 connections were just due to propagation. Not sure. Anyway, I'm seeing connection counts as I would expect now that I've killed the old server. Thanks! Jerry On Thu, Oct 24, 2019 at 3:52 PM Jerry Hebert wrote: > Hey all, > > I've upgraded one of my ensembles to 3.5.5 now > (3.5.5-390fe37ea45dee01bf87dc1c042b5e3dcce88653, specifically). All of my > metrics appeared to be healthy but after the migration, I noticed that the > new ensemble has *all 5 nodes reporting a connection count of 1* (via the > "stat" 4ltr command as well as zk_num_alive_connections from the "mntr" > output). > > The servers are clearly receiving traffic: I can see node counts going up > and down and I can see clients making changes to various keys. I can also > monitor netstat for 2181 connections and again see connections fluctuating > per usual but I still see "Connections: 1" in stats. This translates into > our Datadog agent reporting connections as 1 too. I've been reading through > the code to try to understand how this may be possible but it's a bit of a > slog as I'm unfamiliar with it and I've found myself digging into Netty now. > > I pasted a couple of possibly relevant log lines below. In particular, > note that "secure" is false here and I noticed that the conx count is split > in the code depending on whether or not you're in secure mode. I also find > it odd that I'm seeing 0:0:0:0:0:0:0:0 in the logs which looks like ipv6 to > me and I'm using ipv4 (or at least, I partially am...). I also don't > understand the zxid expectation mismatch. > > 2019-10-24 22:12:08,411 [myid:9] - WARN > [QuorumPeer[myid=9](plain=/0:0:0:0:0:0:0:0:2181)(secure=disabled):Follower@125] > - Got zxid 0x2f00000001 expected 0x1 > > 2019-10-24 22:11:57,349 [myid:9] - INFO [main:ServerCnxnFactory@135] - > Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection > factory > > Any advice would be greatly appreciated. I don't feel comfortable leaving > this server as-is given that it's misreporting connections. Something is > definitely wrong. > > Thanks in advance! > > Jerry > > --000000000000d628120595b04695--