From user-return-11621-archive-asf-public=cust-asf.ponee.io@zookeeper.apache.org Fri Aug 17 03:03:06 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 95C3C180675 for ; Fri, 17 Aug 2018 03:03:05 +0200 (CEST) Received: (qmail 6834 invoked by uid 500); 17 Aug 2018 01:03:04 -0000 Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@zookeeper.apache.org Delivered-To: mailing list user@zookeeper.apache.org Received: (qmail 5684 invoked by uid 99); 17 Aug 2018 01:03:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Aug 2018 01:03:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id E3592C101D for ; Fri, 17 Aug 2018 01:03:02 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.891 X-Spam-Level: *** X-Spam-Status: No, score=3.891 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, KAM_BADIPHTTP=2, NORMAL_HTTP_TO_IP=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id P0qtaQymg8rM for ; Fri, 17 Aug 2018 01:03:01 +0000 (UTC) Received: from mail-io0-f180.google.com (mail-io0-f180.google.com [209.85.223.180]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id DF5B65F178 for ; Fri, 17 Aug 2018 01:03:00 +0000 (UTC) Received: by mail-io0-f180.google.com with SMTP id y24-v6so1849264iob.12 for ; Thu, 16 Aug 2018 18:03:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=wFmDKtTv6hgxvYGMvU8paYQrebhN8ntlBGrhdCthFO8=; b=tYlIMZ+mnr03HZUA6EfeGQKYRWYW9ULVKak7hTHqQbBIwokeW25yneyyLI6khylj1T nJeOzYkRJo7Bj9Iuywcrdxg01R4qfT3X/W0YLgas65oX+fyuMDh19NNxnV9KARJA+loa SlOmTQSnET+UxKpFaaVjExV7Vv9VsPSsi2cOADAwWNb38L0L3eEo6UB8JE1EgLk3Lt1o GaWA/nL5Zq6O7XPrPiQNLKVjx/6UZFQn6xI6A/BC62L+OyiCArTtuV06jSuA0/Xp+ghU Q4dFQEai7apoNEmAXJ7ebGdRF9wkpU5L/KkYq0JHIs6+7ViMfhbSt55r0k57E+KofUdD 2Ssw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=wFmDKtTv6hgxvYGMvU8paYQrebhN8ntlBGrhdCthFO8=; b=hyB3yISbeHygpF2KZFSifOfvsZpE8tAv+xNpP+d0pfHX4Q3iNoyGESXbGIdJWMnJVH GZeB9qYBEFQWZHACVHeBrf663siCAtKTJXDNHx8cxHSa1gOso9tFnVY+96/9rR6qgvAU xjhSNWPq6zFJqJiPqliKUApVgbRR+AcVxcIVYzQfW0V9jnpBtgxnpEUnxElAysHUDlvE 9sG4DG5E64+SFBsGbgk12EY0Qd797TiIYDLS5kCDEadZHyxFYFM+mLFIaxvmd9EYN1yz MpnghIU0eDPwiEaJRVFN8zoYLNMf08A+TUwwU9Nc4RVfyCs+kpT3zIsL3YcxgmnLah3H S0ww== X-Gm-Message-State: AOUpUlETY1sULtMRPQifoPFNgomyKouQkYxJN98+0CsZtuWdyuEhH57o fRpvTHHLk0glJNeqs1q0Z+JHYVAJxpOiQk7db27m3AxyCXw= X-Google-Smtp-Source: AA+uWPycpaUu/P5qp39jFHEXEe/l42x4hIBkJtFRxQg0HG39xtkb2BE315yUWkVS1inyVMiOHjcdLMBzKCUQzjXQomM= X-Received: by 2002:a6b:4f11:: with SMTP id d17-v6mr28145320iob.176.1534467780272; Thu, 16 Aug 2018 18:03:00 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a4f:506:0:0:0:0:0 with HTTP; Thu, 16 Aug 2018 18:02:59 -0700 (PDT) From: Raghav Date: Thu, 16 Aug 2018 18:02:59 -0700 Message-ID: Subject: Please help: Zookeeper not coming up after power down To: Users , user@zookeeper.apache.org Content-Type: multipart/alternative; boundary="000000000000f669b10573971eb3" --000000000000f669b10573971eb3 Content-Type: text/plain; charset="UTF-8" Hi Our 3 node Zookeeper ensemble got powered down, and upon powering up the zookeeper could get quorum and kept throwing these errors. As a result our Kafka cluster was unusable. What is the best way to revive ZK cluster in such situations ? Please suggest. 2018-08-17_00:59:18.87009 2018-08-17 00:59:18,869 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@584] - Cannot open channel to 2 at election address /1.1.1.143:3888 2018-08-17_00:59:18.87011 java.net.ConnectException: Connection refused 2018-08-17_00:59:18.87011 at java.net.PlainSocketImpl.socketConnect(Native Method) 2018-08-17_00:59:18.87011 at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345) 2018-08-17_00:59:18.87012 at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) 2018-08-17_00:59:18.87012 at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) 2018-08-17_00:59:18.87013 at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) 2018-08-17_00:59:18.87013 at java.net.Socket.connect(Socket.java:589) 2018-08-17_00:59:18.87013 at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:558) 2018-08-17_00:59:18.87014 at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:610) 2018-08-17_00:59:18.87014 at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:838) 2018-08-17_00:59:18.87014 at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:957) 2018-08-17_00:59:18.87034 2018-08-17 00:59:18,870 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumPeer$QuorumServer@184] - Resolved hostname: 1.1.1.143 to address: /1.1.1.143 2018-08-17_00:59:18.87095 2018-08-17 00:59:18,870 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@584] - Cannot open channel to 3 at election address /1.1.1.144:3888 2018-08-17_00:59:18.87097 java.net.ConnectException: Connection refused 2018-08-17_00:59:18.87097 at java.net.PlainSocketImpl.socketConnect(Native Method) 2018-08-17_00:59:18.87097 at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345) 2018-08-17_00:59:18.87098 at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) 2018-08-17_00:59:18.87098 at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) 2018-08-17_00:59:18.87098 at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) 2018-08-17_00:59:18.87098 at java.net.Socket.connect(Socket.java:589) 2018-08-17_00:59:18.87099 at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:558) 2018-08-17_00:59:18.87099 at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:610) 2018-08-17_00:59:18.87099 at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:838) 2018-08-17_00:59:18.87099 at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:957) Thanks. R --000000000000f669b10573971eb3--