Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 6AAD2200C40 for ; Thu, 23 Mar 2017 16:03:48 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 69316160B6F; Thu, 23 Mar 2017 15:03:48 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 14DC5160B75 for ; Thu, 23 Mar 2017 16:03:46 +0100 (CET) Received: (qmail 20574 invoked by uid 500); 23 Mar 2017 15:03:41 -0000 Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@zookeeper.apache.org Delivered-To: mailing list user@zookeeper.apache.org Received: (qmail 20563 invoked by uid 99); 23 Mar 2017 15:03:40 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Mar 2017 15:03:40 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 574B71809E2 for ; Thu, 23 Mar 2017 15:03:40 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.001 X-Spam-Level: X-Spam-Status: No, score=-0.001 tagged_above=-999 required=6.31 tests=[RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id qo_SvmeOdWnY for ; Thu, 23 Mar 2017 15:03:35 +0000 (UTC) Received: from mo01c.lpsvcs.com (mo01c.lpsvcs.com [206.201.66.78]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 997FF5FB40 for ; Thu, 23 Mar 2017 15:03:34 +0000 (UTC) X-Disclaimer: Default Received: from unknown (HELO JTCLPSWEXMBH03.lpsvcs.com) ([10.48.131.213]) by jtc-smtp02.lpsvcs.com with ESMTP/TLS/DHE-RSA-AES256-SHA; 23 Mar 2017 11:03:29 -0400 Received: from JTCLPSWEXMBH04.lpsvcs.com ([fe80::7588:4c36:d3fe:edd6]) by JTCLPSWEXMBH03.lpsvcs.com ([fe80::412c:8cb8:20ee:38ab%18]) with mapi id 14.03.0266.001; Thu, 23 Mar 2017 11:03:26 -0400 From: "Cantrell, Curtis" To: "user@zookeeper.apache.org" Subject: Leader throws continuous exceptions when starting a Follower Thread-Topic: Leader throws continuous exceptions when starting a Follower Thread-Index: AdKj5jeTYnrt2tIsTL6QKKukFbNf2Q== Date: Thu, 23 Mar 2017 15:03:26 +0000 Message-ID: <58541717BBD8834B8AA650F2108759222F08E051@JTCLPSWEXMBH04.lpsvcs.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.48.117.218] Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable archived-at: Thu, 23 Mar 2017 15:03:48 -0000 We have a 5 server farm. This behavior has happened before, and it has br= ought our servers to their knees... But today I was manually working and= made it happen. = We are running zookeeper-3.5.2-alpha on Red Hat Enterprise Linux Server rel= ease 6.8 All five are running. #1 Following #2. Following #3. Following #4. LEADING #5. Following 2017-03-23 09:26:40 I stoped #1 with /bin/zkServer.sh stop to adjust the= log levels in log4j.properties 2017-03-23 09:31:47 I started #1 with /bin/zkServer.sh start = All the Logs went CRAZY... ( I have added the Log snippets) = It was not until I stopped #1 again, that the logs settled back down. = I tried to Start #1 again.... The same thing happened. All the logs went c= razy, so I stopped #1 Seeing that the Leader was saying GOODBYE over and over, I decided to stop = the Leader. = I stopped #4 with /bin/zkServer.sh stop.. Everything seemed fine. At thi= s point we have 3 of the 5 zookeepers running. = I started #4 with /bin/zkServer.sh start. Everything was fine. #4 entere= d the ensemble. = I went back and started #1. This time, #1 entered the ensemble as normal. What in the world can cause the entire ensemble go into the infinite kind o= f error condition.. Below are the parts of each log when I attempted to restart #1 after it bei= ng shut down for only 3 minutes. = #04 LEADER LOGS At Stratup of #1 The Leader logged this over and over 1000's of times. = 2017-03-23 09:31:47,441 [myid:4] - INFO [jtcprflegwap04/10.49.129.7:49602:= QuorumCnxManager$Listener@637] - Received connection request /10.49.129.4:4= 4018 2017-03-23 09:31:47,462 [myid:4] - INFO [WorkerReceiver[myid=3D4]:FastLead= erElection@688] - Notification: 2 (message format version), 1 (n.leader), 0= x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x375 (n.peerEPoc= h), LEADING (my state)36f00000000 (n.config version) 2017-03-23 09:31:47,501 [myid:4] - INFO [LearnerHandler-/10.49.129.4:38020= :LearnerHandler@382] - Follower sid: 1 : info : jtcprflegwap01:49601:49602:= participant 2017-03-23 09:31:47,503 [myid:4] - INFO [LearnerHandler-/10.49.129.4:38020= :LearnerHandler@683] - Synchronizing with Follower sid: 1 maxCommittedLog= =3D0x3750000007d minCommittedLog=3D0x37400000001 lastProcessedZxid=3D0x3750= 000007d peerLastZxid=3D0x0 2017-03-23 09:31:47,506 [myid:4] - WARN [LearnerHandler-/10.49.129.4:38020= :LearnerHandler@619] - ******* GOODBYE /10.49.129.4:38020 ******** 2017-03-23 09:31:47,512 [myid:4] - WARN [LearnerHandler-/10.49.129.4:38020= :ZooKeeperThread@55] - Exception occured from thread LearnerHandler-/10.49.= 129.4:38020 java.lang.NullPointerException at org.apache.zookeeper.server.ZKDatabase.calculateTxnLogSizeLimit(ZKDatab= ase.java:279) at org.apache.zookeeper.server.quorum.LearnerHandler.syncFollower(LearnerH= andler.java:753) at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.ja= va:427) 2017-03-23 09:31:47,527 [myid:4] - INFO [WorkerReceiver[myid=3D4]:FastLead= erElection@688] - Notification: 2 (message format version), 1 (n.leader), 0= x0 (n.zxid), 0x0 (n.round), LOOKING (n.state), 1 (n.sid), 0x375 (n.peerEPoc= h), LEADING (my state)36f00000000 (n.config version) 2017-03-23 09:31:47,537 [myid:4] - INFO [LearnerHandler-/10.49.129.4:38022= :LearnerHandler@382] - Follower sid: 1 : info : jtcprflegwap01:49601:49602:= participant 2017-03-23 09:31:47,539 [myid:4] - INFO [LearnerHandler-/10.49.129.4:38022= :LearnerHandler@683] - Synchronizing with Follower sid: 1 maxCommittedLog= =3D0x3750000007d minCommittedLog=3D0x37400000001 lastProcessedZxid=3D0x3750= 000007d peerLastZxid=3D0x0 2017-03-23 09:31:47,540 [myid:4] - WARN [LearnerHandler-/10.49.129.4:38022= :LearnerHandler@619] - ******* GOODBYE /10.49.129.4:38022 ******** 2017-03-23 09:31:47,542 [myid:4] - WARN [LearnerHandler-/10.49.129.4:38022= :ZooKeeperThread@55] - Exception occured from thread LearnerHandler-/10.49.= 129.4:38022 java.lang.NullPointerException at org.apache.zookeeper.server.ZKDatabase.calculateTxnLogSizeLimit(ZKDatab= ase.java:279) at org.apache.zookeeper.server.quorum.LearnerHandler.syncFollower(LearnerH= andler.java:753) at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.ja= va:427) 2017-03-23 09:31:47,551 [myid:4] - INFO [WorkerReceiver[myid=3D4]:FastLead= erElection@688] - Notification: 2 (message format version), 1 (n.leader), 0= x0 (n.zxid), 0x0 (n.round), LOOKING (n.state), 1 (n.sid), 0x375 (n.peerEPoc= h), LEADING (my state)36f00000000 (n.config version) 2017-03-23 09:31:47,563 [myid:4] - INFO [LearnerHandler-/10.49.129.4:38024= :LearnerHandler@382] - Follower sid: 1 : info : jtcprflegwap01:49601:49602:= participant 2017-03-23 09:31:47,566 [myid:4] - INFO [LearnerHandler-/10.49.129.4:38024= :LearnerHandler@683] - Synchronizing with Follower sid: 1 maxCommittedLog= =3D0x3750000007d minCommittedLog=3D0x37400000001 lastProcessedZxid=3D0x3750= 000007d peerLastZxid=3D0x0 2017-03-23 09:31:47,567 [myid:4] - WARN [LearnerHandler-/10.49.129.4:38024= :LearnerHandler@619] - ******* GOODBYE /10.49.129.4:38024 ******** 2017-03-23 09:31:47,568 [myid:4] - WARN [LearnerHandler-/10.49.129.4:38024= :ZooKeeperThread@55] - Exception occured from thread LearnerHandler-/10.49.= 129.4:38024 java.lang.NullPointerException at org.apache.zookeeper.server.ZKDatabase.calculateTxnLogSizeLimit(ZKDatab= ase.java:279) at org.apache.zookeeper.server.quorum.LearnerHandler.syncFollower(LearnerH= andler.java:753) at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.ja= va:427) #02 FOLLOWER LOGS At Stratup of #1 A single connection request and then 1000's of FastLeaderElection messges 2017-03-23 09:31:47,571 [myid:2] - INFO [jtcprflegwap02/10.49.129.5:49602:= QuorumCnxManager$Listener@637] - Received connection request /10.49.129.4:4= 5670 2017-03-23 09:31:47,600 [myid:2] - INFO [WorkerReceiver[myid=3D2]:FastLead= erElection@688] - Notification: 2 (message format version), 1 (n.leader), 0= x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x375 (n.peerEPoc= h), FOLLOWING (my state)36f00000000 (n.config version) 2017-03-23 09:31:47,667 [myid:2] - INFO [WorkerReceiver[myid=3D2]:FastLead= erElection@688] - Notification: 2 (message format version), 1 (n.leader), 0= x0 (n.zxid), 0x0 (n.round), LOOKING (n.state), 1 (n.sid), 0x375 (n.peerEPoc= h), FOLLOWING (my state)36f00000000 (n.config version) 2017-03-23 09:31:47,690 [myid:2] - INFO [WorkerReceiver[myid=3D2]:FastLead= erElection@688] - Notification: 2 (message format version), 1 (n.leader), 0= x0 (n.zxid), 0x0 (n.round), LOOKING (n.state), 1 (n.sid), 0x375 (n.peerEPoc= h), FOLLOWING (my state)36f00000000 (n.config version) #03 FOLLOWER LOGS At Stratup of #1 A single connection request and then 1000's of FastLeaderElection messges 2017-03-23 09:31:47,558 [myid:3] - INFO [jtcprflegwap03/10.49.129.6:49602:= QuorumCnxManager$Listener@637] - Received connection request /10.49.129.4:4= 0956 2017-03-23 09:31:47,575 [myid:3] - INFO [WorkerReceiver[myid=3D3]:FastLead= erElection@688] - Notification: 2 (message format version), 1 (n.leader), 0= x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x375 (n.peerEPoc= h), FOLLOWING (my state)36f00000000 (n.config version) 2017-03-23 09:31:47,647 [myid:3] - INFO [WorkerReceiver[myid=3D3]:FastLead= erElection@688] - Notification: 2 (message format version), 1 (n.leader), 0= x0 (n.zxid), 0x0 (n.round), LOOKING (n.state), 1 (n.sid), 0x375 (n.peerEPoc= h), FOLLOWING (my state)36f00000000 (n.config version) #05 FOLLOWER LOGS At Stratup of #1 A single connection request and then 1000's of FastLeaderElection messages 2017-03-23 09:31:47,467 [myid:5] - INFO [jtcprflegwdm01/10.49.193.109:4960= 2:QuorumCnxManager$Listener@637] - Received connection request /10.49.129.4= :57884 2017-03-23 09:31:47,486 [myid:5] - INFO [WorkerReceiver[myid=3D5]:FastLead= erElection@688] - Notification: 2 (message format version), 1 (n.leader), 0= x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x375 (n.peerEPoc= h), FOLLOWING (my state)36f00000000 (n.config version) 2017-03-23 09:31:47,548 [myid:5] - INFO [WorkerReceiver[myid=3D5]:FastLead= erElection@688] - Notification: 2 (message format version), 1 (n.leader), 0= x0 (n.zxid), 0x0 (n.round), LOOKING (n.state), 1 (n.sid), 0x375 (n.peerEPoc= h), FOLLOWING (my state)36f00000000 (n.config version) #01 FOLLOWING Again after STARTING it up.... As soon as it started following the Leader, an Exception occurred and it di= sconnected.. only to do it again.. 1000's of times. = 2017-03-23 09:31:47,365 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):ZooKeeperServer@156] - Created server= with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir = /opt/eg/zookeeper/tranlog/version-2 snapdir /opt/eg/zookeeper/data/version-2 2017-03-23 09:31:47,367 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):Follower@66] - FOLLOWING - LEADER ELE= CTION TOOK - 20 MS 2017-03-23 09:31:47,399 [myid:1] - WARN [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):Follower@93] - Exception when followi= ng the leader java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacke= t.java:83) at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:9= 9) at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:155) at org.apache.zookeeper.server.quorum.Learner.syncWithLeader(Learner.java:= 361) at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:= 86) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1111) 2017-03-23 09:31:47,406 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):MBeanRegistry@128] - Unregister MBean= [org.apache.ZooKeeperService:name0=3DReplicatedServer_id1,name1=3Dreplica.= 1,name2=3DFollower] 2017-03-23 09:31:47,407 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):Follower@198] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1115) 2017-03-23 09:31:47,408 [myid:1] - WARN [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):QuorumPeer@1158] - PeerState set to L= OOKING 2017-03-23 09:31:47,408 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):QuorumPeer@1033] - LOOKING 2017-03-23 09:31:47,409 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):FastLeaderElection@894] - New electio= n. My id =3D 1, proposed zxid=3D0x0 2017-03-23 09:31:47,410 [myid:1] - INFO [WorkerReceiver[myid=3D1]:FastLead= erElection@688] - Notification: 2 (message format version), 1 (n.leader), 0= x0 (n.zxid), 0x0 (n.round), LOOKING (n.state), 1 (n.sid), 0x375 (n.peerEPoc= h), LOOKING (my state)36f00000000 (n.config version) 2017-03-23 09:31:47,415 [myid:1] - INFO [WorkerReceiver[myid=3D1]:FastLead= erElection@688] - Notification: 2 (message format version), 4 (n.leader), 0= x37400000020 (n.zxid), 0xffffffffffffffff (n.round), FOLLOWING (n.state), 2= (n.sid), 0x375 (n.peerEPoch), LOOKING (my state)36f00000000 (n.config vers= ion) 2017-03-23 09:31:47,416 [myid:1] - INFO [WorkerReceiver[myid=3D1]:FastLead= erElection@688] - Notification: 2 (message format version), 4 (n.leader), 0= x37400000020 (n.zxid), 0xffffffffffffffff (n.round), LEADING (n.state), 4 (= n.sid), 0x375 (n.peerEPoch), LOOKING (my state)36f00000000 (n.config versio= n) 2017-03-23 09:31:47,416 [myid:1] - INFO [WorkerReceiver[myid=3D1]:FastLead= erElection@688] - Notification: 2 (message format version), 4 (n.leader), 0= x37400000020 (n.zxid), 0xffffffffffffffff (n.round), FOLLOWING (n.state), 5= (n.sid), 0x375 (n.peerEPoch), LOOKING (my state)36f00000000 (n.config vers= ion) 2017-03-23 09:31:47,417 [myid:1] - INFO [WorkerReceiver[myid=3D1]:FastLead= erElection@688] - Notification: 2 (message format version), 4 (n.leader), 0= x37400000020 (n.zxid), 0xffffffffffffffff (n.round), FOLLOWING (n.state), 3= (n.sid), 0x375 (n.peerEPoch), LOOKING (my state)36f00000000 (n.config vers= ion) 2017-03-23 09:31:47,418 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):MBeanRegistry@128] - Unregister MBean= [org.apache.ZooKeeperService:name0=3DReplicatedServer_id1,name1=3Dreplica.= 1,name2=3DLeaderElection] 2017-03-23 09:31:47,418 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):QuorumPeer@1109] - FOLLOWING 2017-03-23 09:31:47,418 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):ZooKeeperServer@858] - minSessionTime= out set to 4000 2017-03-23 09:31:47,418 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):ZooKeeperServer@867] - maxSessionTime= out set to 40000 2017-03-23 09:31:47,419 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):ZooKeeperServer@156] - Created server= with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir = /opt/eg/zookeeper/tranlog/version-2 snapdir /opt/eg/zookeeper/data/version-2 2017-03-23 09:31:47,419 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):Follower@66] - FOLLOWING - LEADER ELE= CTION TOOK - 1 MS 2017-03-23 09:31:47,427 [myid:1] - WARN [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):Follower@93] - Exception when followi= ng the leader java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacke= t.java:83) at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:9= 9) at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:155) at org.apache.zookeeper.server.quorum.Learner.syncWithLeader(Learner.java:= 361) at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:= 86) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1111) 2017-03-23 09:31:47,429 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):MBeanRegistry@128] - Unregister MBean= [org.apache.ZooKeeperService:name0=3DReplicatedServer_id1,name1=3Dreplica.= 1,name2=3DFollower] 2017-03-23 09:31:47,429 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):Follower@198] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1115) 2017-03-23 09:31:47,430 [myid:1] - WARN [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):QuorumPeer@1158] - PeerState set to L= OOKING 2017-03-23 09:31:47,430 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):QuorumPeer@1033] - LOOKING 2017-03-23 09:31:47,431 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):FastLeaderElection@894] - New electio= n. My id =3D 1, proposed zxid=3D0x0 2017-03-23 09:31:47,434 [myid:1] - INFO [WorkerReceiver[myid=3D1]:FastLead= erElection@688] - Notification: 2 (message format version), 1 (n.leader), 0= x0 (n.zxid), 0x0 (n.round), LOOKING (n.state), 1 (n.sid), 0x375 (n.peerEPoc= h), LOOKING (my state)36f00000000 (n.config version) 2017-03-23 09:31:47,439 [myid:1] - INFO [WorkerReceiver[myid=3D1]:FastLead= erElection@688] - Notification: 2 (message format version), 4 (n.leader), 0= x37400000020 (n.zxid), 0xffffffffffffffff (n.round), FOLLOWING (n.state), 2= (n.sid), 0x375 (n.peerEPoch), LOOKING (my state)36f00000000 (n.config vers= ion) 2017-03-23 09:31:47,441 [myid:1] - INFO [WorkerReceiver[myid=3D1]:FastLead= erElection@688] - Notification: 2 (message format version), 4 (n.leader), 0= x37400000020 (n.zxid), 0xffffffffffffffff (n.round), LEADING (n.state), 4 (= n.sid), 0x375 (n.peerEPoch), LOOKING (my state)36f00000000 (n.config versio= n) 2017-03-23 09:31:47,443 [myid:1] - INFO [WorkerReceiver[myid=3D1]:FastLead= erElection@688] - Notification: 2 (message format version), 4 (n.leader), 0= x37400000020 (n.zxid), 0xffffffffffffffff (n.round), FOLLOWING (n.state), 3= (n.sid), 0x375 (n.peerEPoch), LOOKING (my state)36f00000000 (n.config vers= ion) 2017-03-23 09:31:47,444 [myid:1] - INFO [WorkerReceiver[myid=3D1]:FastLead= erElection@688] - Notification: 2 (message format version), 4 (n.leader), 0= x37400000020 (n.zxid), 0xffffffffffffffff (n.round), FOLLOWING (n.state), 5= (n.sid), 0x375 (n.peerEPoch), FOLLOWING (my state)36f00000000 (n.config ve= rsion) 2017-03-23 09:31:47,444 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):MBeanRegistry@128] - Unregister MBean= [org.apache.ZooKeeperService:name0=3DReplicatedServer_id1,name1=3Dreplica.= 1,name2=3DLeaderElection] 2017-03-23 09:31:47,445 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):QuorumPeer@1109] - FOLLOWING 2017-03-23 09:31:47,445 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):ZooKeeperServer@858] - minSessionTime= out set to 4000 2017-03-23 09:31:47,445 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):ZooKeeperServer@867] - maxSessionTime= out set to 40000 2017-03-23 09:31:47,446 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):ZooKeeperServer@156] - Created server= with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir = /opt/eg/zookeeper/tranlog/version-2 snapdir /opt/eg/zookeeper/data/version-2 2017-03-23 09:31:47,446 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):Follower@66] - FOLLOWING - LEADER ELE= CTION TOOK - 1 MS 2017-03-23 09:31:47,454 [myid:1] - WARN [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):Follower@93] - Exception when followi= ng the leader java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacke= t.java:83) at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:9= 9) at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:155) at org.apache.zookeeper.server.quorum.Learner.syncWithLeader(Learner.java:= 361) at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:= 86) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1111) 2017-03-23 09:31:47,456 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):MBeanRegistry@128] - Unregister MBean= [org.apache.ZooKeeperService:name0=3DReplicatedServer_id1,name1=3Dreplica.= 1,name2=3DFollower] 2017-03-23 09:31:47,457 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):Follower@198] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1115) 2017-03-23 09:31:47,457 [myid:1] - WARN [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):QuorumPeer@1158] - PeerState set to L= OOKING 2017-03-23 09:31:47,457 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):QuorumPeer@1033] - LOOKING 2017-03-23 09:31:47,458 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):FastLeaderElection@894] - New electio= n. My id =3D 1, proposed zxid=3D0x0 2017-03-23 09:31:47,462 [myid:1] - INFO [WorkerReceiver[myid=3D1]:FastLead= erElection@688] - Notification: 2 (message format version), 1 (n.leader), 0= x0 (n.zxid), 0x0 (n.round), LOOKING (n.state), 1 (n.sid), 0x375 (n.peerEPoc= h), LOOKING (my state)36f00000000 (n.config version) 2017-03-23 09:31:47,464 [myid:1] - INFO [WorkerReceiver[myid=3D1]:FastLead= erElection@688] - Notification: 2 (message format version), 4 (n.leader), 0= x37400000020 (n.zxid), 0xffffffffffffffff (n.round), FOLLOWING (n.state), 2= (n.sid), 0x375 (n.peerEPoch), LOOKING (my state)36f00000000 (n.config vers= ion) 2017-03-23 09:31:47,465 [myid:1] - INFO [WorkerReceiver[myid=3D1]:FastLead= erElection@688] - Notification: 2 (message format version), 4 (n.leader), 0= x37400000020 (n.zxid), 0xffffffffffffffff (n.round), FOLLOWING (n.state), 5= (n.sid), 0x375 (n.peerEPoch), LOOKING (my state)36f00000000 (n.config vers= ion) 2017-03-23 09:31:47,466 [myid:1] - INFO [WorkerReceiver[myid=3D1]:FastLead= erElection@688] - Notification: 2 (message format version), 4 (n.leader), 0= x37400000020 (n.zxid), 0xffffffffffffffff (n.round), FOLLOWING (n.state), 3= (n.sid), 0x375 (n.peerEPoch), LOOKING (my state)36f00000000 (n.config vers= ion) 2017-03-23 09:31:47,467 [myid:1] - INFO [WorkerReceiver[myid=3D1]:FastLead= erElection@688] - Notification: 2 (message format version), 4 (n.leader), 0= x37400000020 (n.zxid), 0xffffffffffffffff (n.round), LEADING (n.state), 4 (= n.sid), 0x375 (n.peerEPoch), LOOKING (my state)36f00000000 (n.config versio= n) 2017-03-23 09:31:47,467 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):MBeanRegistry@128] - Unregister MBean= [org.apache.ZooKeeperService:name0=3DReplicatedServer_id1,name1=3Dreplica.= 1,name2=3DLeaderElection] 2017-03-23 09:31:47,468 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):QuorumPeer@1109] - FOLLOWING 2017-03-23 09:31:47,468 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):ZooKeeperServer@858] - minSessionTime= out set to 4000 2017-03-23 09:31:47,468 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):ZooKeeperServer@867] - maxSessionTime= out set to 40000 2017-03-23 09:31:47,468 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):ZooKeeperServer@156] - Created server= with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir = /opt/eg/zookeeper/tranlog/version-2 snapdir /opt/eg/zookeeper/data/version-2 2017-03-23 09:31:47,468 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):Follower@66] - FOLLOWING - LEADER ELE= CTION TOOK - 0 MS 2017-03-23 09:31:47,476 [myid:1] - WARN [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):Follower@93] - Exception when followi= ng the leader java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacke= t.java:83) at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:9= 9) at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:155) at org.apache.zookeeper.server.quorum.Learner.syncWithLeader(Learner.java:= 361) at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:= 86) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1111) 2017-03-23 09:31:47,477 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):MBeanRegistry@128] - Unregister MBean= [org.apache.ZooKeeperService:name0=3DReplicatedServer_id1,name1=3Dreplica.= 1,name2=3DFollower] 2017-03-23 09:31:47,477 [myid:1] - INFO [QuorumPeer[myid=3D1](plain=3D/0:0= :0:0:0:0:0:0:2181)(secure=3Ddisabled):Follower@198] - shutdown called java.lang.Exception: shutdown Follower at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:198) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1115) Thank you, Curtis Cantrell The information contained in this message is proprietary and/or confidentia= l. If you are not the intended recipient, please: (i) delete the message an= d all copies; (ii) do not disclose, distribute or use the message in any ma= nner; and (iii) notify the sender immediately. In addition, please be aware= that any message addressed to our domain is subject to archiving and revie= w by persons other than the intended recipient. Thank you.