zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Hunt <ph...@apache.org>
Subject Re: regarding zookeper cluster setup replication, config issues and inconsistent state
Date Mon, 16 May 2016 22:14:08 GMT
You might want to check out the "zk-smoketest" tool that I created some
time ago. It's specifically made to verify correct ensemble setup and
quorum formation:
https://github.com/phunt/zk-smoketest

Patrick

On Sat, May 14, 2016 at 6:10 AM, Flavio Junqueira <fpj@apache.org> wrote:

> If there is an active leader, then the ensemble is in sync. The simplest
> way to check if the ensemble is working is to use zkCli to connect and
> perform a few operations. You can also use four-letter commands to to check
> the health of the servers.
>
> I didn't understand the point about a follower following. Perhaps it was
> waiting for its connection to the leader to time out?
>
> -Flavio
>
> > On 14 May 2016, at 02:16, R Krishna <krishna81m@gmail.com> wrote:
> >
> > Hmm, we are using the same version as 3.4.5 with similar explanation as
> > this bug. Bad experience for a first time setup.
> > https://issues.apache.org/jira/browse/ZOOKEEPER-1653
> >
> > So I finally stop ZooKeeper, clear all the data except the myid files as
> > follows:
> >
> > Clean restart all three servers one by one:
> > find /var/lib/zookeeper -name "*"
> > cat /var/lib/zookeeper/myid
> > rm -r /var/lib/zookeeper/ve*
> >> /var/lib/zookeeper/zookeeper_server.pid
> > ls -ltrh /var/lib/zookeeper/
> >
> > Q.) Everything seemed fine except on one of the followers I noticed
> > following. What does this mean?
> > Q.) How do I know the zookeeper cluster is stable and the data is in
> sync?:
> >
> > 2016-05-14 00:31:18,424 - WARN
> > [RecvWorker:2:QuorumCnxManager$RecvWorker@762] - Connection broken for
> id
> > 2, my id = 1, error =
> > java.io.EOFException
> >        at java.io.DataInputStream.readInt(DataInputStream.java:392)
> >        at
> >
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:747)
> > 2016-05-14 00:31:18,424 - WARN
> > [RecvWorker:2:QuorumCnxManager$RecvWorker@765] - Interrupting SendWorker
> > 2016-05-14 00:31:18,425 - WARN
> > [SendWorker:2:QuorumCnxManager$SendWorker@679] - Interrupted while
> waiting
> > for message on queue
> > java.lang.InterruptedException
> >        at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> >        at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> >        at
> > java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> >        at
> >
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:831)
> >        at
> >
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:62)
> >        at
> >
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:667)
> > 2016-05-14 00:31:18,425 - WARN
> > [SendWorker:2:QuorumCnxManager$SendWorker@688] - Send worker leaving
> thread
> >
> >
> > More logs:
> > kafka@awo-p05-kafk01:~$ echo stat | nc X.Y.Z.75 2181
> > Zookeeper version: 3.4.5--1, built on 06/10/2013 17:26 GMT
> > Clients:
> > /X.Y.Z.77:33995[1](queued=0,recved=214,sent=216)
> > /X.Y.Z.75:44172[0](queued=0,recved=1,sent=0)
> >
> > Latency min/avg/max: 0/0/19
> > Received: 239
> > Sent: 240
> > Connections: 2
> > Outstanding: 0
> > Zxid: 0x100000021
> > Mode: follower
> > Node count: 19
> > 2016-05-14 01:14:24,892 - INFO  [Thread-19:NIOServerCnxn@1001] - Closed
> > socket connection for client /X.Y.Z.75:44172 (no session established for
> > client)
> > 2016-05-14 01:14:24,892 - INFO  [Thread-19:NIOServerCnxn@1001] - Closed
> > socket connection for client /X.Y.Z.75:44172 (no session established for
> > client)
> > kafka@awo-p05-kafk01:~$ echo stat | nc X.Y.Z.76 2181
> > echo stat | nc X.Y.Z.77 2181Zookeeper version: 3.4.5--1, built on
> > 06/10/2013 17:26 GMT
> > Clients:
> > /X.Y.Z.75:53076[0](queued=0,recved=1,sent=0)
> >
> > Latency min/avg/max: 0/1/8
> > Received: 36
> > Sent: 35
> > Connections: 1
> > Outstanding: 0
> > Zxid: 0x100000021
> > Mode: follower
> > Node count: 19
> > kafka@awo-p05-kafk01:~$ echo stat | nc X.Y.Z.77 2181
> > Zookeeper version: 3.4.5--1, built on 06/10/2013 17:26 GMT
> > Clients:
> > /X.Y.Z.75:55941[0](queued=0,recved=1,sent=0)
> >
> > Latency min/avg/max: 0/1/3
> > Received: 27
> > Sent: 26
> > Connections: 1
> > Outstanding: 0
> > Zxid: 0x100000021
> > Mode: leader
> > Node count: 19
> >
> >
> >
> > On Fri, May 13, 2016 at 12:59 PM, R Krishna <krishna81m@gmail.com>
> wrote:
> >
> >> As I said before, I cannot even restart one server, it automatically
> >> brings up another process.
> >>
> >> I tried specifically setting the PID.
> >> ps -aef | grep -i zoo
> >> vim /var/lib/zookeeper/zookeeper_server.pid
> >> sudo /usr/share/zookeeper/bin/zkServer.sh restart
> >>
> >> or stop, neither works. Is there a setting to shutdown zookeper and
> bring
> >> up one by one in 3 node cluster?
> >>
> >>
> >> On Fri, May 13, 2016 at 12:57 PM, R Krishna <krishna81m@gmail.com>
> wrote:
> >>
> >>> I have a fairly simple config file (below), I tried to reboot the
> machine
> >>> but server 75 never restarts properly by exposing LISTEN port on 3888
> and
> >>> obviously get 2016-05-13 12:54:58,555 - WARN
> >>> [WorkerSender[myid=3]:QuorumCnxManager@368] - Cannot open channel to 1
> >>> at election address /172.28.84.75:3888. Whereas 75 is unable to expose
> >>> 3888 and unable to connect to other servers with those exceptions shown
> >>> before.
> >>>
> >>> Yes, I chose a distinct id=1 to 3 for each server. How do you do a
> >>> rolling restart? and where do you specify to take it easy if it cannot
> find
> >>> all servers?
> >>>
> >>> # The number of milliseconds of each tick
> >>> tickTime=2000
> >>> # The number of ticks that the initial
> >>> # synchronization phase can take
> >>> initLimit=10
> >>> # The number of ticks that can pass between
> >>> # sending a request and getting an acknowledgement
> >>> syncLimit=5
> >>> # the directory where the snapshot is stored.
> >>> dataDir=/var/lib/zookeeper
> >>> # Place the dataLogDir to a separate physical disc for better
> performance
> >>> # dataLogDir=/disk2/zookeeper
> >>>
> >>> # the port at which the clients will connect
> >>> clientPort=2181
> >>>
> >>> # specify all zookeeper servers
> >>> # The fist port is used by followers to connect to the leader
> >>> # The second one is used for leader election
> >>> server.1=X.Y.Z.75:2888:3888
> >>> server.2=X.Y.Z.76:2888:3888
> >>> server.3=X.Y.Z.98:2888:3888
> >>>
> >>>
> >>> On Fri, May 13, 2016 at 3:51 AM, Flavio Junqueira <fpj@apache.org>
> wrote:
> >>>
> >>>> Hi there,
> >>>>
> >>>> The myid needs to contain the id for each server in the ensemble, so
> >>>> each server will have a distinct value in its myid file.
> >>>>
> >>>> The problem might be with you configuration file. I think you say that
> >>>> you have specified the servers in the config file of each server, but
> >>>> perhaps you want to have a look at the documentation to see if there
> is
> >>>> anything you're missing. If you're not sure, please post it here.
> >>>>
> >>>> In the 3.4 branch of ZK, you have to do a rolling upgrade of the
> servers.
> >>>>
> >>>> -Flavio
> >>>>
> >>>>> On 13 May 2016, at 11:15, R Krishna <krishna81m@gmail.com>
wrote:
> >>>>>
> >>>>> Just tried to setup a 2 zookeeper cluster for the first time one
each
> >>>> for
> >>>>> my 2 Kafka broker cluster and came across following issues:
> >>>>> 1. Do we have to specify a separate value in vim
> >>>> ./var/lib/zookeeper/myid
> >>>>> although they are separate machine instances?
> >>>>> 2. I kept seeing Mode:standalone between the two servers although
I
> saw
> >>>>> connectivity between these two. After restarts, I saw them go to
> >>>>> Follower/Leader.
> >>>>> /usr/share/zookeeper/bin/zkServer.sh status
> >>>>>   JMX enabled by default
> >>>>>   Using config: /etc/zookeeper/conf/zoo.cfg
> >>>>>   Mode: standalone
> >>>>> 3. The data was completely inconsistent, I was able to connect to
> each
> >>>> one
> >>>>> run the all netcat status commands from the other server without
any
> >>>> issue.
> >>>>> However, Kafka broker data was inconsistent and kept failing, is
> there
> >>>> a
> >>>>> way to confirm if both nodes are in sync and part of same cluster?
> >>>>> org.I0Itec.zkclient.exception.ZkNoNodeException:
> >>>>> org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode =
> >>>>> NoNode for /config/changes
> >>>>>
> >>>>> 4. Whenever I updated the .cfg file, I cannot do a sudo
> >>>>> /usr/share/zookeeper/bin/zkServer.sh restart, I have to force kill
> the
> >>>> pid,
> >>>>> in which case in brings up another process reading the latest .cfg,
> >>>> why is
> >>>>> this so?
> >>>>>
> >>>>> 5. I realized we need at least 3 to make an ensemble, so I created
> and
> >>>>> added another ZK host updated the .cfg and force killed the process
> so
> >>>> it
> >>>>> reads the latest config and started getting these exceptions. Yes,
> this
> >>>>> probably means I have run out of connections.
> >>>>>
> >>>>> *And finally, how do I safely restart such a cluster when adding
new
> >>>> nodes
> >>>>> and then force them to sync data?*
> >>>>>
> >>>>> MASTER: 75: ::::::::::::::::::::::::::::::::::::
> >>>>> 3 09:56:03,823 - INFO  [main:FileSnap@83] - Reading snapshot
> >>>>> /var/lib/zookeeper/version-2/snapshot.30
> >>>>> 2016-05-13 09:56:03,860 - ERROR [main:FileTxnSnapLog@210] - Parent
> >>>>> /brokers/ids missing for /brokers/ids/2
> >>>>> 2016-05-13 09:56:03,862 - ERROR [main:QuorumPeer@453] - Unable to
> load
> >>>>> database on disk
> >>>>> java.io.IOException: Failed to process transaction type: 1 error:
> >>>>> KeeperErrorCode = NoNode for /brokers/ids
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:153)
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:417)
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:409)
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:151)
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111)
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
> >>>>> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> >>>>> KeeperErrorCode = NoNode for /brokers/ids
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.processTransaction(FileTxnSnapLog.java:211)
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:151)
> >>>>>       ... 6 more
> >>>>> 2016-05-13 09:56:03,865 - ERROR [main:QuorumPeerMain@89] -
> Unexpected
> >>>>> exception, exiting abnormally
> >>>>> java.lang.RuntimeException: Unable to run quorum server
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:454)
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:409)
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:151)
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111)
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
> >>>>> Caused by: java.io.IOException: Failed to process transaction type:
1
> >>>>> error: KeeperErrorCode = NoNode for /brokers/ids
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:153)
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:417)
> >>>>>       ... 4 more
> >>>>> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> >>>>> KeeperErrorCode = NoNode for /brokers/ids
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.processTransaction(FileTxnSnapLog.java:211)
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:151)
> >>>>>       ... 6 more
> >>>>>
> >>>>>
> >>>>> 2016-05-13 09:57:29,084 - ERROR [main:QuorumPeerMain@89] -
> Unexpected
> >>>>> exception, exiting abnormally
> >>>>> java.lang.RuntimeException: Unable to run quorum server
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:454)
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:409)
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:151)
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111)
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
> >>>>> Caused by: java.io.IOException: Failed to process transaction type:
1
> >>>>> error: KeeperErrorCode = NoNode for /brokers/ids
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:153)
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:417)
> >>>>>       ... 4 more
> >>>>> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> >>>>> KeeperErrorCode = NoNode for /brokers/ids
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.processTransaction(FileTxnSnapLog.java:211)
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:151)
> >>>>>       ... 6 more
> >>>>>
> >>>>>
> >>>>> SECOND: 76
> >>>> :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
> >>>>> ING (n.state), 3 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
> >>>>> 2016-05-13 09:42:40,650 - WARN
> >>>>> [RecvWorker:1:QuorumCnxManager$RecvWorker@762] - Connection broken
> >>>> for id
> >>>>> 1, my id = 2, error =
> >>>>> java.io.EOFException
> >>>>>       at java.io.DataInputStream.readInt(DataInputStream.java:392)
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:747)
> >>>>> 2016-05-13 09:42:40,650 - WARN
> >>>>> [RecvWorker:1:QuorumCnxManager$RecvWorker@765] - Interrupting
> >>>> SendWorker
> >>>>> 2016-05-13 09:42:40,651 - WARN
> >>>>> [SendWorker:1:QuorumCnxManager$SendWorker@679] - Interrupted while
> >>>> waiting
> >>>>> for message on queue
> >>>>> java.lang.InterruptedException
> >>>>>       at
> >>>>>
> >>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> >>>>>       at
> >>>>>
> >>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> >>>>>       at
> >>>>>
> >>>>
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:831)
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:62)
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:667)
> >>>>> 2016-05-13 09:42:40,651 - WARN
> >>>>> [SendWorker:1:QuorumCnxManager$SendWorker@688] - Send worker leaving
> >>>> threa
> >>>>>
> >>>>>
> >>>>> ..... then these ...............
> >>>>>
> >>>>> ==> /var/log/zookeeper/zookeeper.log <==
> >>>>> 2016-05-13 10:01:20,334 - INFO  [NIOServerCxn.Factory:
> >>>>> 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket
> >>>> connection
> >>>>> from /X.Y.Z.75:58954
> >>>>> 2016-05-13 10:01:20,334 - WARN  [NIOServerCxn.Factory:
> >>>>> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@354] - Exception causing close
of
> >>>>> session 0x0 due to java.io.IOException: ZooKeeperServer not running
> >>>>> 2016-05-13 10:01:20,335 - INFO  [NIOServerCxn.Factory:
> >>>>> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed socket connection
> >>>> for
> >>>>> client /X.Y.Z.75:58954 (no session established for client)
> >>>>>
> >>>>> ==> /home/kafka/kafka/kafka.log <==
> >>>>> [2016-05-13 10:01:20,412] INFO Opening socket connection to server
> >>>>> X.Y.Z.75/X.Y.Z.75:2181. Will not attempt to authenticate using SASL
> >>>>> (unknown error) (org.apache.zookeeper.ClientCnxn)
> >>>>> [2016-05-13 10:01:20,413] INFO Socket connection established to
> >>>>> X.Y.Z.75/X.Y.Z.75:2181, initiating session
> >>>> (org.apache.zookeeper.ClientCnxn)
> >>>>> [2016-05-13 10:01:20,637] WARN Session 0x254a9245fc00000 for server
> >>>>> X.Y.Z.75/X.Y.Z.75:2181, unexpected error, closing socket connection
> and
> >>>>> attempting reconnect (org.apache.zookeeper.ClientCnxn)
> >>>>> java.io.IOException: Connection reset by peer
> >>>>>       at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> >>>>>       at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> >>>>>       at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> >>>>>       at sun.nio.ch.IOUtil.read(IOUtil.java:192)
> >>>>>       at
> sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:384)
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
> >>>>>       at
> >>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
> >>>>> [2016-05-13 10:01:21,782] INFO Opening socket connection to server
> >>>>> X.Y.Z.76/X.Y.Z.76:2181. Will not attempt to authenticate using SASL
> >>>>> (unknown error) (org.apache.zookeeper.ClientCnxn)
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> THIRD - added last::::::::::::::::::::::::::::::::::::::::
> >>>>>
> >>>>> LOWING (n.state), 2 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
> >>>>> 2016-05-13 03:03:39,540 - INFO
> >>>>> [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@774]
-
> >>>>> Notification time out: 25600
> >>>>> 2016-05-13 03:03:39,569 - WARN
> >>>> [WorkerSender[myid=3]:QuorumCnxManager@368]
> >>>>> - Cannot open channel to 1 at election address /X.Y.Z.75:3888
> >>>>> java.net.ConnectException: Connection refused
> >>>>>       at java.net.PlainSocketImpl.socketConnect(Native Method)
> >>>>>       at
> >>>>>
> >>>>
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> >>>>>       at
> >>>>>
> >>>>
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> >>>>>       at
> >>>>>
> >>>>
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> >>>>>       at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> >>>>>       at java.net.Socket.connect(Socket.java:579)
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:354)
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:327)
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:393)
> >>>>>       at
> >>>>>
> >>>>
> org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:365)
> >>>>>       at java.lang.Thread.run(Thread.java:745)
> >>>>> 2016-05-13 03:03:39,570 - INFO
> >>>>> [WorkerReceiver[myid=3]:FastLeaderElection@542] - Notification:
3
> >>>>> (n.leader), 0x100000052 (n.zxid), 0x108d2 (n.round), LOOKING
> >>>> (n.state), 3
> >>>>> (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
> >>>>> 2016-05-13 03:03:39,596 - INFO
> >>>>> [WorkerReceiver[myid=3]:FastLeaderElection@542] - Notification:
3
> >>>>> (n.leader), 0x100000052 (n.zxid), 0x108d1 (n.round), FOLLOWING
> >>>> (n.state), 2
> >>>>> (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
> >>>>> 2016-05-13 03:03:47,801 - INFO
> >>>>> [WorkerReceiver[myid=3]:FastLeaderElection@542] - Notification:
2
> >>>>> (n.leader), 0x100000052 (n.zxid), 0x108d2 (n.round), LOOKING
> >>>> (n.state), 2
> >>>>> (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
> >>>>> 2016-05-13 03:03:48,013 - INFO
> >>>>> [WorkerReceiver[myid=3]:FastLeaderElection@542] - Notification:
2
> >>>>> (n.leader), 0x100000052 (n.zxid), 0x108d2 (n.round), LOOKING
> >>>> (n.state), 2
> >>>>> (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
> >>>>> 2016-05-13 03:03:48,415 - INFO
> >>>>> [WorkerReceiver[myid=3]:FastLeaderElection@542] - Notification:
2
> >>>>> (n.leader), 0x100000052 (n.zxid), 0x108d2 (n.round), LOOKING
> >>>> (n.state), 2
> >>>>> (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
> >>>>> 2016-05-13 03:03:49,216 - INFO
> >>>>> [WorkerReceiver[myid=3]:FastLeaderElection@542] - Notification:
2
> >>>>> (n.leader), 0x100000052 (n.zxid), 0x108d2 (n.round), LOOKING
> >>>> (n.state), 2
> >>>>> (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
> >>>>> 2016-05-13 03:03:50,818 - INFO
> >>>>> [WorkerReceiver[myid=3]:FastLeaderElection@542] - Notification:
2
> >>>>> (n.leader), 0x100000052 (n.zxid), 0x108d2 (n.round), LOOKING
> >>>> (n.state), 2
> >>>>> (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> Radha Krishna, Proddaturi
> >>> 253-234-5657
> >>>
> >>
> >>
> >>
> >> --
> >> Radha Krishna, Proddaturi
> >> 253-234-5657
> >>
> >
> >
> >
> > --
> > Radha Krishna, Proddaturi
> > 253-234-5657
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message