1. Ids/Nodes are correct and their is no problem,
2. ZooKeeper started on all three nodes within a span of 5 seconds.
3. Our cluster is not on AWS or on any cloud for that matter.
Is there any zookeeper settings that could possibly help in such scenarios ?
Thanks
R
On Thu, Aug 16, 2018 at 6:12 PM, Dan Simoes <dan@tcell.io> wrote:
> Ensure ids/nodes are correct in zoo.cfg and zookeeper is running on each.
> Also any changes to ports being open? If it’s aws, beck security groups.
> Node 1 cannot talk to the other two nodes.
>
> > On Aug 16, 2018, at 6:02 PM, Raghav <raghavastic@gmail.com> wrote:
> >
> > Hi
> >
> > Our 3 node Zookeeper ensemble got powered down, and upon powering up the
> > zookeeper could get quorum and kept throwing these errors. As a result
> our
> > Kafka cluster was unusable. What is the best way to revive ZK cluster in
> > such situations ? Please suggest.
> >
> >
> > 2018-08-17_00:59:18.87009 2018-08-17 00:59:18,869 [myid:1] - WARN
> > [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@584] - Cannot
> > open channel to 2 at election address /1.1.1.143:3888
> > 2018-08-17_00:59:18.87011 java.net.ConnectException: Connection refused
> > 2018-08-17_00:59:18.87011 at
> > java.net.PlainSocketImpl.socketConnect(Native Method)
> > 2018-08-17_00:59:18.87011 at
> > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:
> 345)
> > 2018-08-17_00:59:18.87012 at
> > java.net.AbstractPlainSocketImpl.connectToAddress(
> AbstractPlainSocketImpl.java:206)
> > 2018-08-17_00:59:18.87012 at
> > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:
> 188)
> > 2018-08-17_00:59:18.87013 at
> > java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> > 2018-08-17_00:59:18.87013 at java.net.Socket.connect(
> Socket.java:589)
> > 2018-08-17_00:59:18.87013 at
> > org.apache.zookeeper.server.quorum.QuorumCnxManager.
> connectOne(QuorumCnxManager.java:558)
> > 2018-08-17_00:59:18.87014 at
> > org.apache.zookeeper.server.quorum.QuorumCnxManager.
> connectAll(QuorumCnxManager.java:610)
> > 2018-08-17_00:59:18.87014 at
> > org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(
> FastLeaderElection.java:838)
> > 2018-08-17_00:59:18.87014 at
> > org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:957)
> > 2018-08-17_00:59:18.87034 2018-08-17 00:59:18,870 [myid:1] - INFO
> > [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumPeer$QuorumServer@184] -
> > Resolved hostname: 1.1.1.143 to address: /1.1.1.143
> > 2018-08-17_00:59:18.87095 2018-08-17 00:59:18,870 [myid:1] - WARN
> > [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@584] - Cannot
> > open channel to 3 at election address /1.1.1.144:3888
> > 2018-08-17_00:59:18.87097 java.net.ConnectException: Connection refused
> > 2018-08-17_00:59:18.87097 at
> > java.net.PlainSocketImpl.socketConnect(Native Method)
> > 2018-08-17_00:59:18.87097 at
> > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:
> 345)
> > 2018-08-17_00:59:18.87098 at
> > java.net.AbstractPlainSocketImpl.connectToAddress(
> AbstractPlainSocketImpl.java:206)
> > 2018-08-17_00:59:18.87098 at
> > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:
> 188)
> > 2018-08-17_00:59:18.87098 at
> > java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> > 2018-08-17_00:59:18.87098 at java.net.Socket.connect(
> Socket.java:589)
> > 2018-08-17_00:59:18.87099 at
> > org.apache.zookeeper.server.quorum.QuorumCnxManager.
> connectOne(QuorumCnxManager.java:558)
> > 2018-08-17_00:59:18.87099 at
> > org.apache.zookeeper.server.quorum.QuorumCnxManager.
> connectAll(QuorumCnxManager.java:610)
> > 2018-08-17_00:59:18.87099 at
> > org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(
> FastLeaderElection.java:838)
> > 2018-08-17_00:59:18.87099 at
> > org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:957)
> >
> > Thanks.
> >
> > R
>
--
Raghav
|