From user-return-11456-archive-asf-public=cust-asf.ponee.io@zookeeper.apache.org Mon May 14 08:41:19 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 4569B180627 for ; Mon, 14 May 2018 08:41:19 +0200 (CEST) Received: (qmail 87294 invoked by uid 500); 14 May 2018 06:41:17 -0000 Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@zookeeper.apache.org Delivered-To: mailing list user@zookeeper.apache.org Received: (qmail 87193 invoked by uid 99); 14 May 2018 06:41:15 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 May 2018 06:41:15 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 748EAC0514 for ; Mon, 14 May 2018 06:41:15 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.287 X-Spam-Level: *** X-Spam-Status: No, score=3.287 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, KAM_BADIPHTTP=2, NORMAL_HTTP_TO_IP=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.604, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id ngWl5G-Rlr6P for ; Mon, 14 May 2018 06:41:14 +0000 (UTC) Received: from mail-io0-f170.google.com (mail-io0-f170.google.com [209.85.223.170]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 267A35F178 for ; Mon, 14 May 2018 06:41:14 +0000 (UTC) Received: by mail-io0-f170.google.com with SMTP id t23-v6so13889844ioc.10 for ; Sun, 13 May 2018 23:41:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=2AypgTqhYhtLbZEkAOJYuqnZNp5Ii7ZSH4FcfLnOnyo=; b=qaP2tVbgz8F8mrkEHB77ClobPcvGqxyKNI5ovO9/QnPc7ldBkxNO3U7Di/visvxUDz BazOfNaHNkzaYWA4zv1NRJzsohZoHXpYM9hudY9BH8K+Or0z9AJFrnMgx2JrQwP3Vjcj 9MU2ucenPE+bDhELTkZdUGsANlGGDjsx4EEzIeDnZiQR7I4CpR2Z4qJxpBhwsVjao2bc YfI/BKfv+xlQx7ciRg+IvcFKnYHRBfTu9rznNuMweHNhTSVBakrSLTKkKtYeSG3Uo4RD vYqMfXs9o/PslKzWl34FANOILmVYhwXmBiMYyW7MsmM6ghmC/AU05P5Q3fn3x21pUKDK B6DQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=2AypgTqhYhtLbZEkAOJYuqnZNp5Ii7ZSH4FcfLnOnyo=; b=lgCeV9vlI7vOMZ9Rrkz8wfYht5fcXbHJjbW91lWEdksIxnunT31Km8u3LV4AXZX8sb ipjDa9Z5F4+kjP05rr9UqPaAPuIRmE8CfLq/vnCJv+tsH/EerIDm6ATGCbqwYBQONzpy nf1iySFPSTfkMvgaBjQ7Ak74zRdSoXHmn85sggcE2PndWgXeshn7lMmJr4s1wZ/FiXLl GJaQKcBhoBccxyXKZ/LVI1S7EsAuRKy4eHcOscBGLr1anrQptO3jgbbSnl3CFnJaJaS4 uD/tXfwiay+/EbY6mJt4kNrQAHmQzYw1w9yHbx1YIDi7zZJqSawgvGhl4AsW9L17Jxsf i5cg== X-Gm-Message-State: ALKqPweS3ooPo/aUA2QBRBNAc1V8nU9GHEdzMiDRz25n7s8xvzSQvwuo 8EhzVITWXlqMZ9/aNaUhobZES4m8AjKWLqXYiQU= X-Google-Smtp-Source: AB8JxZp5qOUWIpktgmuTZ62fZWFUMXeKcedGeGhTafWajkBISxEFGkIFJushdQT8ls+b5u7mApwhnMhF/SwEylZF6XE= X-Received: by 2002:a6b:1604:: with SMTP id 4-v6mr9757953iow.147.1526280066687; Sun, 13 May 2018 23:41:06 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a4f:cbc6:0:0:0:0:0 with HTTP; Sun, 13 May 2018 23:41:06 -0700 (PDT) In-Reply-To: References: From: Raghav Date: Sun, 13 May 2018 23:41:06 -0700 Message-ID: Subject: Re: Help Needed: Leadership Issue upon ZK Restart (ZooKeeper 3.4.9) To: user@zookeeper.apache.org Content-Type: multipart/alternative; boundary="00000000000033e8ae056c24c51f" --00000000000033e8ae056c24c51f Content-Type: text/plain; charset="UTF-8" Yes, it seems reachable. After the logs that I pasted, it seems to have printed logs that show it is connected. Anything else I could be missing ? Thanks. On Fri, May 11, 2018 at 4:35 PM, Prasanth Mathialagan < prasanthmathialagan@gmail.com> wrote: > Is 1.1.1.143:3888 reachable from the host in which you see this error? > > On Fri, May 11, 2018 at 3:11 PM, Raghav wrote: > > > Hi > > > > We have a 3 node zk ensemble as well as 3 node Kafka Cluster. They both > are > > hosted on the same 3 VMs. > > > > Before Restart > > 1. We were on Kafka 0.10.2.1 > > > > After Restart > > 1. We moved to Kafka 1.1 > > > > We observe that Kafkas report leadership issues, and for lot of > partitions > > Leader is -1. I see some logs in ZK that mainly point towards some > > connectivity issue around restart time. > > > > *We are stuck on this one for a while now, and neither rolling restart of > > ZK is helping. Can you please help or point us how we can debug this.* > > > > *2018-05-11_17:20:49.00305 2018-05-11 17:20:49,002 [myid:1] - INFO > > [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 > (message > > format version), 1 (n.leader), 0x200000112 (n.zxid), 0x1 (n.round), > LOOKING > > (n.state), 1 (n.sid), 0x2 (n.peerEpoch) LOOKING (my > > state) 2018-05-11_17:20:49.01201 > > 2018-05-11 17:20:49,010 [myid:1] - WARN > > [WorkerSender[myid=1]:QuorumCnxManager@400] - Cannot open channel to 2 > at > > election address /1.1.1.143:3888 > > > > 2018-05-11_17:20:49.01203 java.net.ConnectException: Connection > > refused > > 2018-05-11_17:20:49.01203 at > > java.net.PlainSocketImpl.socketConnect(Native > > Method) > > 2018-05-11_17:20:49.01203 at > > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java: > > 345) > > 2018-05-11_17:20:49.01203 at > > java.net.AbstractPlainSocketImpl.connectToAddress( > > AbstractPlainSocketImpl.java:206) > > 2018-05-11_17:20:49.01204 at > > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java: > 188) > > 2018-05-11_17:20:49.01204 at > > java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > > 2018-05-11_17:20:49.01204 at > > java.net.Socket.connect(Socket.java:589) > > 2018-05-11_17:20:49.01204 at > > org.apache.zookeeper.server.quorum.QuorumCnxManager. > > connectOne(QuorumCnxManager.java:381) > > 2018-05-11_17:20:49.01204 at > > org.apache.zookeeper.server.quorum.QuorumCnxManager. > > toSend(QuorumCnxManager.java:354) > > 2018-05-11_17:20:49.01205 at > > org.apache.zookeeper.server.quorum.FastLeaderElection$ > > Messenger$WorkerSender.process(FastLeaderElection.java:452) > > 2018-05-11_17:20:49.01205 at > > org.apache.zookeeper.server.quorum.FastLeaderElection$ > > Messenger$WorkerSender.run(FastLeaderElection.java:433) > > 2018-05-11_17:20:49.01206 at java.lang.Thread.run(Thread. > java:745)* > > > > > > Raghav > > > -- Raghav --00000000000033e8ae056c24c51f--