zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Junqueira <...@apache.org>
Subject Re: regarding zookeper cluster setup replication, config issues and inconsistent state
Date Sat, 14 May 2016 13:10:08 GMT
If there is an active leader, then the ensemble is in sync. The simplest way to check if the
ensemble is working is to use zkCli to connect and perform a few operations. You can also
use four-letter commands to to check the health of the servers.

I didn't understand the point about a follower following. Perhaps it was waiting for its connection
to the leader to time out?

-Flavio

> On 14 May 2016, at 02:16, R Krishna <krishna81m@gmail.com> wrote:
> 
> Hmm, we are using the same version as 3.4.5 with similar explanation as
> this bug. Bad experience for a first time setup.
> https://issues.apache.org/jira/browse/ZOOKEEPER-1653
> 
> So I finally stop ZooKeeper, clear all the data except the myid files as
> follows:
> 
> Clean restart all three servers one by one:
> find /var/lib/zookeeper -name "*"
> cat /var/lib/zookeeper/myid
> rm -r /var/lib/zookeeper/ve*
>> /var/lib/zookeeper/zookeeper_server.pid
> ls -ltrh /var/lib/zookeeper/
> 
> Q.) Everything seemed fine except on one of the followers I noticed
> following. What does this mean?
> Q.) How do I know the zookeeper cluster is stable and the data is in sync?:
> 
> 2016-05-14 00:31:18,424 - WARN
> [RecvWorker:2:QuorumCnxManager$RecvWorker@762] - Connection broken for id
> 2, my id = 1, error =
> java.io.EOFException
>        at java.io.DataInputStream.readInt(DataInputStream.java:392)
>        at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:747)
> 2016-05-14 00:31:18,424 - WARN
> [RecvWorker:2:QuorumCnxManager$RecvWorker@765] - Interrupting SendWorker
> 2016-05-14 00:31:18,425 - WARN
> [SendWorker:2:QuorumCnxManager$SendWorker@679] - Interrupted while waiting
> for message on queue
> java.lang.InterruptedException
>        at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
>        at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
>        at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
>        at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:831)
>        at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:62)
>        at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:667)
> 2016-05-14 00:31:18,425 - WARN
> [SendWorker:2:QuorumCnxManager$SendWorker@688] - Send worker leaving thread
> 
> 
> More logs:
> kafka@awo-p05-kafk01:~$ echo stat | nc X.Y.Z.75 2181
> Zookeeper version: 3.4.5--1, built on 06/10/2013 17:26 GMT
> Clients:
> /X.Y.Z.77:33995[1](queued=0,recved=214,sent=216)
> /X.Y.Z.75:44172[0](queued=0,recved=1,sent=0)
> 
> Latency min/avg/max: 0/0/19
> Received: 239
> Sent: 240
> Connections: 2
> Outstanding: 0
> Zxid: 0x100000021
> Mode: follower
> Node count: 19
> 2016-05-14 01:14:24,892 - INFO  [Thread-19:NIOServerCnxn@1001] - Closed
> socket connection for client /X.Y.Z.75:44172 (no session established for
> client)
> 2016-05-14 01:14:24,892 - INFO  [Thread-19:NIOServerCnxn@1001] - Closed
> socket connection for client /X.Y.Z.75:44172 (no session established for
> client)
> kafka@awo-p05-kafk01:~$ echo stat | nc X.Y.Z.76 2181
> echo stat | nc X.Y.Z.77 2181Zookeeper version: 3.4.5--1, built on
> 06/10/2013 17:26 GMT
> Clients:
> /X.Y.Z.75:53076[0](queued=0,recved=1,sent=0)
> 
> Latency min/avg/max: 0/1/8
> Received: 36
> Sent: 35
> Connections: 1
> Outstanding: 0
> Zxid: 0x100000021
> Mode: follower
> Node count: 19
> kafka@awo-p05-kafk01:~$ echo stat | nc X.Y.Z.77 2181
> Zookeeper version: 3.4.5--1, built on 06/10/2013 17:26 GMT
> Clients:
> /X.Y.Z.75:55941[0](queued=0,recved=1,sent=0)
> 
> Latency min/avg/max: 0/1/3
> Received: 27
> Sent: 26
> Connections: 1
> Outstanding: 0
> Zxid: 0x100000021
> Mode: leader
> Node count: 19
> 
> 
> 
> On Fri, May 13, 2016 at 12:59 PM, R Krishna <krishna81m@gmail.com> wrote:
> 
>> As I said before, I cannot even restart one server, it automatically
>> brings up another process.
>> 
>> I tried specifically setting the PID.
>> ps -aef | grep -i zoo
>> vim /var/lib/zookeeper/zookeeper_server.pid
>> sudo /usr/share/zookeeper/bin/zkServer.sh restart
>> 
>> or stop, neither works. Is there a setting to shutdown zookeper and bring
>> up one by one in 3 node cluster?
>> 
>> 
>> On Fri, May 13, 2016 at 12:57 PM, R Krishna <krishna81m@gmail.com> wrote:
>> 
>>> I have a fairly simple config file (below), I tried to reboot the machine
>>> but server 75 never restarts properly by exposing LISTEN port on 3888 and
>>> obviously get 2016-05-13 12:54:58,555 - WARN
>>> [WorkerSender[myid=3]:QuorumCnxManager@368] - Cannot open channel to 1
>>> at election address /172.28.84.75:3888. Whereas 75 is unable to expose
>>> 3888 and unable to connect to other servers with those exceptions shown
>>> before.
>>> 
>>> Yes, I chose a distinct id=1 to 3 for each server. How do you do a
>>> rolling restart? and where do you specify to take it easy if it cannot find
>>> all servers?
>>> 
>>> # The number of milliseconds of each tick
>>> tickTime=2000
>>> # The number of ticks that the initial
>>> # synchronization phase can take
>>> initLimit=10
>>> # The number of ticks that can pass between
>>> # sending a request and getting an acknowledgement
>>> syncLimit=5
>>> # the directory where the snapshot is stored.
>>> dataDir=/var/lib/zookeeper
>>> # Place the dataLogDir to a separate physical disc for better performance
>>> # dataLogDir=/disk2/zookeeper
>>> 
>>> # the port at which the clients will connect
>>> clientPort=2181
>>> 
>>> # specify all zookeeper servers
>>> # The fist port is used by followers to connect to the leader
>>> # The second one is used for leader election
>>> server.1=X.Y.Z.75:2888:3888
>>> server.2=X.Y.Z.76:2888:3888
>>> server.3=X.Y.Z.98:2888:3888
>>> 
>>> 
>>> On Fri, May 13, 2016 at 3:51 AM, Flavio Junqueira <fpj@apache.org> wrote:
>>> 
>>>> Hi there,
>>>> 
>>>> The myid needs to contain the id for each server in the ensemble, so
>>>> each server will have a distinct value in its myid file.
>>>> 
>>>> The problem might be with you configuration file. I think you say that
>>>> you have specified the servers in the config file of each server, but
>>>> perhaps you want to have a look at the documentation to see if there is
>>>> anything you're missing. If you're not sure, please post it here.
>>>> 
>>>> In the 3.4 branch of ZK, you have to do a rolling upgrade of the servers.
>>>> 
>>>> -Flavio
>>>> 
>>>>> On 13 May 2016, at 11:15, R Krishna <krishna81m@gmail.com> wrote:
>>>>> 
>>>>> Just tried to setup a 2 zookeeper cluster for the first time one each
>>>> for
>>>>> my 2 Kafka broker cluster and came across following issues:
>>>>> 1. Do we have to specify a separate value in vim
>>>> ./var/lib/zookeeper/myid
>>>>> although they are separate machine instances?
>>>>> 2. I kept seeing Mode:standalone between the two servers although I saw
>>>>> connectivity between these two. After restarts, I saw them go to
>>>>> Follower/Leader.
>>>>> /usr/share/zookeeper/bin/zkServer.sh status
>>>>>   JMX enabled by default
>>>>>   Using config: /etc/zookeeper/conf/zoo.cfg
>>>>>   Mode: standalone
>>>>> 3. The data was completely inconsistent, I was able to connect to each
>>>> one
>>>>> run the all netcat status commands from the other server without any
>>>> issue.
>>>>> However, Kafka broker data was inconsistent and kept failing, is there
>>>> a
>>>>> way to confirm if both nodes are in sync and part of same cluster?
>>>>> org.I0Itec.zkclient.exception.ZkNoNodeException:
>>>>> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode
=
>>>>> NoNode for /config/changes
>>>>> 
>>>>> 4. Whenever I updated the .cfg file, I cannot do a sudo
>>>>> /usr/share/zookeeper/bin/zkServer.sh restart, I have to force kill the
>>>> pid,
>>>>> in which case in brings up another process reading the latest .cfg,
>>>> why is
>>>>> this so?
>>>>> 
>>>>> 5. I realized we need at least 3 to make an ensemble, so I created and
>>>>> added another ZK host updated the .cfg and force killed the process so
>>>> it
>>>>> reads the latest config and started getting these exceptions. Yes, this
>>>>> probably means I have run out of connections.
>>>>> 
>>>>> *And finally, how do I safely restart such a cluster when adding new
>>>> nodes
>>>>> and then force them to sync data?*
>>>>> 
>>>>> MASTER: 75: ::::::::::::::::::::::::::::::::::::
>>>>> 3 09:56:03,823 - INFO  [main:FileSnap@83] - Reading snapshot
>>>>> /var/lib/zookeeper/version-2/snapshot.30
>>>>> 2016-05-13 09:56:03,860 - ERROR [main:FileTxnSnapLog@210] - Parent
>>>>> /brokers/ids missing for /brokers/ids/2
>>>>> 2016-05-13 09:56:03,862 - ERROR [main:QuorumPeer@453] - Unable to load
>>>>> database on disk
>>>>> java.io.IOException: Failed to process transaction type: 1 error:
>>>>> KeeperErrorCode = NoNode for /brokers/ids
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:153)
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:417)
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:409)
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:151)
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111)
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
>>>>> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
>>>>> KeeperErrorCode = NoNode for /brokers/ids
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.persistence.FileTxnSnapLog.processTransaction(FileTxnSnapLog.java:211)
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:151)
>>>>>       ... 6 more
>>>>> 2016-05-13 09:56:03,865 - ERROR [main:QuorumPeerMain@89] - Unexpected
>>>>> exception, exiting abnormally
>>>>> java.lang.RuntimeException: Unable to run quorum server
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:454)
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:409)
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:151)
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111)
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
>>>>> Caused by: java.io.IOException: Failed to process transaction type: 1
>>>>> error: KeeperErrorCode = NoNode for /brokers/ids
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:153)
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:417)
>>>>>       ... 4 more
>>>>> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
>>>>> KeeperErrorCode = NoNode for /brokers/ids
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.persistence.FileTxnSnapLog.processTransaction(FileTxnSnapLog.java:211)
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:151)
>>>>>       ... 6 more
>>>>> 
>>>>> 
>>>>> 2016-05-13 09:57:29,084 - ERROR [main:QuorumPeerMain@89] - Unexpected
>>>>> exception, exiting abnormally
>>>>> java.lang.RuntimeException: Unable to run quorum server
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:454)
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:409)
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:151)
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111)
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
>>>>> Caused by: java.io.IOException: Failed to process transaction type: 1
>>>>> error: KeeperErrorCode = NoNode for /brokers/ids
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:153)
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:417)
>>>>>       ... 4 more
>>>>> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
>>>>> KeeperErrorCode = NoNode for /brokers/ids
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.persistence.FileTxnSnapLog.processTransaction(FileTxnSnapLog.java:211)
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:151)
>>>>>       ... 6 more
>>>>> 
>>>>> 
>>>>> SECOND: 76
>>>> :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
>>>>> ING (n.state), 3 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
>>>>> 2016-05-13 09:42:40,650 - WARN
>>>>> [RecvWorker:1:QuorumCnxManager$RecvWorker@762] - Connection broken
>>>> for id
>>>>> 1, my id = 2, error =
>>>>> java.io.EOFException
>>>>>       at java.io.DataInputStream.readInt(DataInputStream.java:392)
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:747)
>>>>> 2016-05-13 09:42:40,650 - WARN
>>>>> [RecvWorker:1:QuorumCnxManager$RecvWorker@765] - Interrupting
>>>> SendWorker
>>>>> 2016-05-13 09:42:40,651 - WARN
>>>>> [SendWorker:1:QuorumCnxManager$SendWorker@679] - Interrupted while
>>>> waiting
>>>>> for message on queue
>>>>> java.lang.InterruptedException
>>>>>       at
>>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
>>>>>       at
>>>>> 
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
>>>>>       at
>>>>> 
>>>> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:831)
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:62)
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:667)
>>>>> 2016-05-13 09:42:40,651 - WARN
>>>>> [SendWorker:1:QuorumCnxManager$SendWorker@688] - Send worker leaving
>>>> threa
>>>>> 
>>>>> 
>>>>> ..... then these ...............
>>>>> 
>>>>> ==> /var/log/zookeeper/zookeeper.log <==
>>>>> 2016-05-13 10:01:20,334 - INFO  [NIOServerCxn.Factory:
>>>>> 0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket
>>>> connection
>>>>> from /X.Y.Z.75:58954
>>>>> 2016-05-13 10:01:20,334 - WARN  [NIOServerCxn.Factory:
>>>>> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@354] - Exception causing close of
>>>>> session 0x0 due to java.io.IOException: ZooKeeperServer not running
>>>>> 2016-05-13 10:01:20,335 - INFO  [NIOServerCxn.Factory:
>>>>> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed socket connection
>>>> for
>>>>> client /X.Y.Z.75:58954 (no session established for client)
>>>>> 
>>>>> ==> /home/kafka/kafka/kafka.log <==
>>>>> [2016-05-13 10:01:20,412] INFO Opening socket connection to server
>>>>> X.Y.Z.75/X.Y.Z.75:2181. Will not attempt to authenticate using SASL
>>>>> (unknown error) (org.apache.zookeeper.ClientCnxn)
>>>>> [2016-05-13 10:01:20,413] INFO Socket connection established to
>>>>> X.Y.Z.75/X.Y.Z.75:2181, initiating session
>>>> (org.apache.zookeeper.ClientCnxn)
>>>>> [2016-05-13 10:01:20,637] WARN Session 0x254a9245fc00000 for server
>>>>> X.Y.Z.75/X.Y.Z.75:2181, unexpected error, closing socket connection and
>>>>> attempting reconnect (org.apache.zookeeper.ClientCnxn)
>>>>> java.io.IOException: Connection reset by peer
>>>>>       at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>>>>>       at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>>>>>       at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>>>>>       at sun.nio.ch.IOUtil.read(IOUtil.java:192)
>>>>>       at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:384)
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
>>>>>       at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
>>>>> [2016-05-13 10:01:21,782] INFO Opening socket connection to server
>>>>> X.Y.Z.76/X.Y.Z.76:2181. Will not attempt to authenticate using SASL
>>>>> (unknown error) (org.apache.zookeeper.ClientCnxn)
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> THIRD - added last::::::::::::::::::::::::::::::::::::::::
>>>>> 
>>>>> LOWING (n.state), 2 (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
>>>>> 2016-05-13 03:03:39,540 - INFO
>>>>> [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@774] -
>>>>> Notification time out: 25600
>>>>> 2016-05-13 03:03:39,569 - WARN
>>>> [WorkerSender[myid=3]:QuorumCnxManager@368]
>>>>> - Cannot open channel to 1 at election address /X.Y.Z.75:3888
>>>>> java.net.ConnectException: Connection refused
>>>>>       at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>>       at
>>>>> 
>>>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
>>>>>       at
>>>>> 
>>>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
>>>>>       at
>>>>> 
>>>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
>>>>>       at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>>>>       at java.net.Socket.connect(Socket.java:579)
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:354)
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:327)
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:393)
>>>>>       at
>>>>> 
>>>> org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:365)
>>>>>       at java.lang.Thread.run(Thread.java:745)
>>>>> 2016-05-13 03:03:39,570 - INFO
>>>>> [WorkerReceiver[myid=3]:FastLeaderElection@542] - Notification: 3
>>>>> (n.leader), 0x100000052 (n.zxid), 0x108d2 (n.round), LOOKING
>>>> (n.state), 3
>>>>> (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
>>>>> 2016-05-13 03:03:39,596 - INFO
>>>>> [WorkerReceiver[myid=3]:FastLeaderElection@542] - Notification: 3
>>>>> (n.leader), 0x100000052 (n.zxid), 0x108d1 (n.round), FOLLOWING
>>>> (n.state), 2
>>>>> (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
>>>>> 2016-05-13 03:03:47,801 - INFO
>>>>> [WorkerReceiver[myid=3]:FastLeaderElection@542] - Notification: 2
>>>>> (n.leader), 0x100000052 (n.zxid), 0x108d2 (n.round), LOOKING
>>>> (n.state), 2
>>>>> (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
>>>>> 2016-05-13 03:03:48,013 - INFO
>>>>> [WorkerReceiver[myid=3]:FastLeaderElection@542] - Notification: 2
>>>>> (n.leader), 0x100000052 (n.zxid), 0x108d2 (n.round), LOOKING
>>>> (n.state), 2
>>>>> (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
>>>>> 2016-05-13 03:03:48,415 - INFO
>>>>> [WorkerReceiver[myid=3]:FastLeaderElection@542] - Notification: 2
>>>>> (n.leader), 0x100000052 (n.zxid), 0x108d2 (n.round), LOOKING
>>>> (n.state), 2
>>>>> (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
>>>>> 2016-05-13 03:03:49,216 - INFO
>>>>> [WorkerReceiver[myid=3]:FastLeaderElection@542] - Notification: 2
>>>>> (n.leader), 0x100000052 (n.zxid), 0x108d2 (n.round), LOOKING
>>>> (n.state), 2
>>>>> (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
>>>>> 2016-05-13 03:03:50,818 - INFO
>>>>> [WorkerReceiver[myid=3]:FastLeaderElection@542] - Notification: 2
>>>>> (n.leader), 0x100000052 (n.zxid), 0x108d2 (n.round), LOOKING
>>>> (n.state), 2
>>>>> (n.sid), 0x1 (n.peerEPoch), LOOKING (my state)
>>>> 
>>>> 
>>> 
>>> 
>>> --
>>> Radha Krishna, Proddaturi
>>> 253-234-5657
>>> 
>> 
>> 
>> 
>> --
>> Radha Krishna, Proddaturi
>> 253-234-5657
>> 
> 
> 
> 
> -- 
> Radha Krishna, Proddaturi
> 253-234-5657


Mime
View raw message