zookeeper-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andor Molnar (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (ZOOKEEPER-2486) Flaky Test: org.apache.zookeeper.test.QuorumZxidSyncTest.testBehindLeader
Date Mon, 15 Oct 2018 10:27:03 GMT

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andor Molnar updated ZOOKEEPER-2486:
------------------------------------
        Parent: ZOOKEEPER-3170
    Issue Type: Sub-task  (was: Test)

> Flaky Test: org.apache.zookeeper.test.QuorumZxidSyncTest.testBehindLeader
> -------------------------------------------------------------------------
>
>                 Key: ZOOKEEPER-2486
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2486
>             Project: ZooKeeper
>          Issue Type: Sub-task
>          Components: tests
>    Affects Versions: 3.4.8
>            Reporter: Michael Han
>            Priority: Major
>              Labels: flaky, flaky-test
>             Fix For: 3.5.2
>
>
> From https://builds.apache.org/job/ZooKeeper_branch34_jdk7/1156
> {noformat}
> Error Message
> waiting for server up
> Stacktrace
> junit.framework.AssertionFailedError: waiting for server up
> 	at org.apache.zookeeper.test.QuorumBase.startServers(QuorumBase.java:183)
> 	at org.apache.zookeeper.test.QuorumBase.startServers(QuorumBase.java:113)
> 	at org.apache.zookeeper.test.QuorumZxidSyncTest.testBehindLeader(QuorumZxidSyncTest.java:67)
> 	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:55)
> Standard Output
> 2016-07-21 08:11:45,722 [myid:] - INFO  [main:PortAssignment@32] - assigning port 11221
> 2016-07-21 08:11:45,729 [myid:] - INFO  [main:ZKTestCase$1@50] - STARTING testBehindLeader
> 2016-07-21 08:11:45,729 [myid:] - INFO  [main:QuorumBase@69] - QuorumBase.setup null
> 2016-07-21 08:11:45,926 [myid:] - INFO  [main:PortAssignment@32] - assigning port 11222
> 2016-07-21 08:11:45,926 [myid:] - INFO  [main:PortAssignment@32] - assigning port 11223
> 2016-07-21 08:11:45,927 [myid:] - INFO  [main:PortAssignment@32] - assigning port 11224
> 2016-07-21 08:11:45,927 [myid:] - INFO  [main:PortAssignment@32] - assigning port 11225
> 2016-07-21 08:11:45,927 [myid:] - INFO  [main:PortAssignment@32] - assigning port 11226
> 2016-07-21 08:11:45,928 [myid:] - INFO  [main:PortAssignment@32] - assigning port 11227
> 2016-07-21 08:11:45,928 [myid:] - INFO  [main:PortAssignment@32] - assigning port 11228
> 2016-07-21 08:11:45,928 [myid:] - INFO  [main:PortAssignment@32] - assigning port 11229
> 2016-07-21 08:11:45,928 [myid:] - INFO  [main:PortAssignment@32] - assigning port 11230
> 2016-07-21 08:11:45,929 [myid:] - INFO  [main:PortAssignment@32] - assigning port 11231
> 2016-07-21 08:11:45,929 [myid:] - INFO  [main:QuorumBase@93] - Ports are: 127.0.0.1:11222,127.0.0.1:11223,127.0.0.1:11224,127.0.0.1:11225,127.0.0.1:11226
> 2016-07-21 08:11:45,946 [myid:] - INFO  [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
> 2016-07-21 08:11:45,946 [myid:] - INFO  [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
> 2016-07-21 08:11:45,948 [myid:] - INFO  [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
> 2016-07-21 08:11:45,948 [myid:] - INFO  [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
> 2016-07-21 08:11:45,949 [myid:] - INFO  [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
> 2016-07-21 08:11:45,949 [myid:] - INFO  [main:QuorumBase@142] - creating QuorumPeer 1 port 11222
> 2016-07-21 08:11:45,961 [myid:] - INFO  [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11222
> 2016-07-21 08:11:45,979 [myid:] - INFO  [main:QuorumBase@145] - creating QuorumPeer 2 port 11223
> 2016-07-21 08:11:45,979 [myid:] - INFO  [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11223
> 2016-07-21 08:11:45,980 [myid:] - INFO  [main:QuorumBase@148] - creating QuorumPeer 3 port 11224
> 2016-07-21 08:11:45,980 [myid:] - INFO  [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11224
> 2016-07-21 08:11:45,981 [myid:] - INFO  [main:QuorumBase@151] - creating QuorumPeer 4 port 11225
> 2016-07-21 08:11:45,981 [myid:] - INFO  [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11225
> 2016-07-21 08:11:45,981 [myid:] - INFO  [main:QuorumBase@154] - creating QuorumPeer 5 port 11226
> 2016-07-21 08:11:45,982 [myid:] - INFO  [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:11226
> 2016-07-21 08:11:45,982 [myid:] - INFO  [main:QuorumBase@163] - QuorumPeer 1 voting view: {1=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@47628981, 2=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@163198c4, 3=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1224773e, 4=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@181090c0, 5=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@678f3997}
> 2016-07-21 08:11:45,983 [myid:] - INFO  [main:QuorumBase@164] - QuorumPeer 2 voting view: {1=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@47628981, 2=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@163198c4, 3=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1224773e, 4=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@181090c0, 5=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@678f3997}
> 2016-07-21 08:11:45,983 [myid:] - INFO  [main:QuorumBase@165] - QuorumPeer 3 voting view: {1=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@47628981, 2=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@163198c4, 3=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1224773e, 4=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@181090c0, 5=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@678f3997}
> 2016-07-21 08:11:45,983 [myid:] - INFO  [main:QuorumBase@166] - QuorumPeer 4 voting view: {1=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@47628981, 2=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@163198c4, 3=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1224773e, 4=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@181090c0, 5=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@678f3997}
> 2016-07-21 08:11:45,983 [myid:] - INFO  [main:QuorumBase@167] - QuorumPeer 5 voting view: {1=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@47628981, 2=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@163198c4, 3=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@1224773e, 4=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@181090c0, 5=org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@678f3997}
> 2016-07-21 08:11:45,983 [myid:] - INFO  [main:QuorumBase@169] - start QuorumPeer 1
> 2016-07-21 08:11:45,988 [myid:] - INFO  [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
> 2016-07-21 08:11:45,990 [myid:] - INFO  [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
> 2016-07-21 08:11:45,996 [myid:] - INFO  [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /127.0.0.1:12227
> 2016-07-21 08:11:46,001 [myid:] - INFO  [main:QuorumBase@171] - start QuorumPeer 2
> 2016-07-21 08:11:46,002 [myid:] - INFO  [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
> 2016-07-21 08:11:46,003 [myid:] - INFO  [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
> 2016-07-21 08:11:46,004 [myid:] - INFO  [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /127.0.0.1:12228
> 2016-07-21 08:11:46,006 [myid:] - INFO  [main:QuorumBase@173] - start QuorumPeer 3
> 2016-07-21 08:11:46,006 [myid:] - INFO  [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
> 2016-07-21 08:11:46,007 [myid:] - INFO  [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
> 2016-07-21 08:11:46,007 [myid:] - INFO  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11222:QuorumPeer@774] - LOOKING
> 2016-07-21 08:11:46,008 [myid:] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11223:QuorumPeer@774] - LOOKING
> 2016-07-21 08:11:46,009 [myid:] - INFO  [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /127.0.0.1:12229
> 2016-07-21 08:11:46,009 [myid:] - INFO  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11222:FastLeaderElection@818] - New election. My id =  1, proposed zxid=0x0
> 2016-07-21 08:11:46,009 [myid:] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11223:FastLeaderElection@818] - New election. My id =  2, proposed zxid=0x0
> 2016-07-21 08:11:46,009 [myid:] - INFO  [main:QuorumBase@175] - start QuorumPeer 4
> 2016-07-21 08:11:46,010 [myid:] - INFO  [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
> 2016-07-21 08:11:46,011 [myid:] - INFO  [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
> 2016-07-21 08:11:46,011 [myid:] - INFO  [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11224:QuorumPeer@774] - LOOKING
> 2016-07-21 08:11:46,011 [myid:] - INFO  [/127.0.0.1:12228:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:38924
> 2016-07-21 08:11:46,011 [myid:] - INFO  [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11224:FastLeaderElection@818] - New election. My id =  3, proposed zxid=0x0
> 2016-07-21 08:11:46,011 [myid:] - INFO  [/127.0.0.1:12227:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:43415
> 2016-07-21 08:11:46,011 [myid:] - INFO  [WorkerSender[myid=1]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (2, 1)
> 2016-07-21 08:11:46,011 [myid:] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,013 [myid:] - INFO  [WorkerSender[myid=1]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (3, 1)
> 2016-07-21 08:11:46,014 [myid:] - INFO  [/127.0.0.1:12229:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:51548
> 2016-07-21 08:11:46,014 [myid:] - INFO  [main:QuorumBase@177] - start QuorumPeer 5
> 2016-07-21 08:11:46,015 [myid:] - INFO  [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /127.0.0.1:12230
> 2016-07-21 08:11:46,015 [myid:] - INFO  [main:QuorumPeer@533] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
> 2016-07-21 08:11:46,015 [myid:] - WARN  [WorkerSender[myid=1]:QuorumCnxManager@400] - Cannot open channel to 4 at election address /127.0.0.1:12230
> java.net.ConnectException: Connection refused
> 	at java.net.PlainSocketImpl.socketConnect(Native Method)
> 	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> 	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> 	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> 	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> 	at java.net.Socket.connect(Socket.java:579)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
> 	at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
> 	at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
> 	at java.lang.Thread.run(Thread.java:745)
> 2016-07-21 08:11:46,018 [myid:] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,018 [myid:] - INFO  [/127.0.0.1:12229:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:51551
> 2016-07-21 08:11:46,016 [myid:] - INFO  [/127.0.0.1:12227:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:43417
> 2016-07-21 08:11:46,016 [myid:] - INFO  [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:QuorumPeer@774] - LOOKING
> 2016-07-21 08:11:46,016 [myid:] - INFO  [main:QuorumPeer@548] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
> 2016-07-21 08:11:46,019 [myid:] - INFO  [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:FastLeaderElection@818] - New election. My id =  4, proposed zxid=0x0
> 2016-07-21 08:11:46,019 [myid:] - INFO  [/127.0.0.1:12230:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:46102
> 2016-07-21 08:11:46,018 [myid:] - INFO  [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,018 [myid:] - INFO  [WorkerSender[myid=2]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (3, 2)
> 2016-07-21 08:11:46,018 [myid:] - INFO  [WorkerSender[myid=1]:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
> 2016-07-21 08:11:46,018 [myid:] - INFO  [/127.0.0.1:12228:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:38928
> 2016-07-21 08:11:46,021 [myid:] - INFO  [WorkerSender[myid=2]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (4, 2)
> 2016-07-21 08:11:46,021 [myid:] - INFO  [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,021 [myid:] - INFO  [/127.0.0.1:12229:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:51553
> 2016-07-21 08:11:46,020 [myid:] - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,019 [myid:] - INFO  [WorkerSender[myid=3]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (4, 3)
> 2016-07-21 08:11:46,022 [myid:] - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,021 [myid:] - WARN  [WorkerSender[myid=2]:QuorumCnxManager@400] - Cannot open channel to 5 at election address /127.0.0.1:12231
> java.net.ConnectException: Connection refused
> 	at java.net.PlainSocketImpl.socketConnect(Native Method)
> 	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> 	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> 	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> 	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> 	at java.net.Socket.connect(Socket.java:579)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
> 	at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
> 	at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
> 	at java.lang.Thread.run(Thread.java:745)
> 2016-07-21 08:11:46,021 [myid:] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,021 [myid:] - WARN  [WorkerSender[myid=1]:QuorumCnxManager@400] - Cannot open channel to 5 at election address /127.0.0.1:12231
> java.net.ConnectException: Connection refused
> 	at java.net.PlainSocketImpl.socketConnect(Native Method)
> 	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> 	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> 	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> 	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> 	at java.net.Socket.connect(Socket.java:579)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
> 	at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
> 	at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
> 	at java.lang.Thread.run(Thread.java:745)
> 2016-07-21 08:11:46,023 [myid:] - INFO  [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,023 [myid:] - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,023 [myid:] - INFO  [WorkerSender[myid=2]:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
> 2016-07-21 08:11:46,024 [myid:] - INFO  [/127.0.0.1:12230:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:46104
> 2016-07-21 08:11:46,022 [myid:] - INFO  [ListenerThread:QuorumCnxManager$Listener@534] - My election bind port: /127.0.0.1:12231
> 2016-07-21 08:11:46,022 [myid:] - WARN  [WorkerSender[myid=3]:QuorumCnxManager@400] - Cannot open channel to 5 at election address /127.0.0.1:12231
> java.net.ConnectException: Connection refused
> 	at java.net.PlainSocketImpl.socketConnect(Native Method)
> 	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> 	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> 	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> 	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> 	at java.net.Socket.connect(Socket.java:579)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
> 	at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
> 	at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
> 	at java.lang.Thread.run(Thread.java:745)
> 2016-07-21 08:11:46,025 [myid:] - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,025 [myid:] - INFO  [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,025 [myid:] - INFO  [main:QuorumBase@179] - started QuorumPeer 5
> 2016-07-21 08:11:46,026 [myid:] - INFO  [main:QuorumBase@181] - Checking ports 127.0.0.1:11222,127.0.0.1:11223,127.0.0.1:11224,127.0.0.1:11225,127.0.0.1:11226
> 2016-07-21 08:11:46,025 [myid:] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,025 [myid:] - INFO  [/127.0.0.1:12227:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:43428
> 2016-07-21 08:11:46,027 [myid:] - INFO  [/127.0.0.1:12230:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:46109
> 2016-07-21 08:11:46,025 [myid:] - INFO  [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,024 [myid:] - INFO  [WorkerSender[myid=1]:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
> 2016-07-21 08:11:46,027 [myid:] - INFO  [WorkerSender[myid=4]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (5, 4)
> 2016-07-21 08:11:46,027 [myid:] - INFO  [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,026 [myid:] - INFO  [/127.0.0.1:12228:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:38938
> 2016-07-21 08:11:46,025 [myid:] - INFO  [WorkerSender[myid=3]:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
> 2016-07-21 08:11:46,025 [myid:] - INFO  [WorkerSender[myid=2]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (4, 2)
> 2016-07-21 08:11:46,028 [myid:] - INFO  [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11222
> 2016-07-21 08:11:46,028 [myid:] - WARN  [RecvWorker:2:QuorumCnxManager$RecvWorker@810] - Connection broken for id 2, my id = 4, error = 
> java.net.SocketException: Socket closed
> 	at java.net.SocketInputStream.socketRead0(Native Method)
> 	at java.net.SocketInputStream.read(SocketInputStream.java:152)
> 	at java.net.SocketInputStream.read(SocketInputStream.java:122)
> 	at java.net.SocketInputStream.read(SocketInputStream.java:210)
> 	at java.io.DataInputStream.readInt(DataInputStream.java:387)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:11:46,029 [myid:] - WARN  [RecvWorker:2:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:11:46,028 [myid:] - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,027 [myid:] - INFO  [QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11226:QuorumPeer@774] - LOOKING
> 2016-07-21 08:11:46,027 [myid:] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,027 [myid:] - INFO  [/127.0.0.1:12231:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:52086
> 2016-07-21 08:11:46,030 [myid:] - WARN  [RecvWorker:4:QuorumCnxManager$RecvWorker@810] - Connection broken for id 4, my id = 2, error = 
> java.io.EOFException
> 	at java.io.DataInputStream.readInt(DataInputStream.java:392)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:11:46,030 [myid:] - WARN  [SendWorker:4:QuorumCnxManager$SendWorker@732] - Exception when using channel: for id 4 my id = 2 error = java.net.SocketException: Broken pipe
> 2016-07-21 08:11:46,029 [myid:] - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,031 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory@192] - Accepted socket connection from /127.0.0.1:52363
> 2016-07-21 08:11:46,029 [myid:] - INFO  [/127.0.0.1:12228:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:38940
> 2016-07-21 08:11:46,029 [myid:] - INFO  [WorkerSender[myid=1]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (5, 1)
> 2016-07-21 08:11:46,029 [myid:] - INFO  [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,032 [myid:] - INFO  [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,032 [myid:] - INFO  [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,029 [myid:] - WARN  [SendWorker:2:QuorumCnxManager$SendWorker@732] - Exception when using channel: for id 2 my id = 4 error = java.net.SocketException: Socket closed
> 2016-07-21 08:11:46,033 [myid:] - WARN  [SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:11:46,029 [myid:] - INFO  [WorkerSender[myid=2]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (5, 2)
> 2016-07-21 08:11:46,028 [myid:] - INFO  [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,033 [myid:] - INFO  [WorkerSender[myid=1]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (5, 1)
> 2016-07-21 08:11:46,032 [myid:] - INFO  [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,031 [myid:] - INFO  [/127.0.0.1:12231:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:52088
> 2016-07-21 08:11:46,031 [myid:] - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,031 [myid:] - INFO  [/127.0.0.1:12230:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:46117
> 2016-07-21 08:11:46,030 [myid:] - WARN  [SendWorker:4:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:11:46,030 [myid:] - WARN  [RecvWorker:4:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:11:46,030 [myid:] - INFO  [WorkerSender[myid=3]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (5, 3)
> 2016-07-21 08:11:46,030 [myid:] - INFO  [QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11226:FastLeaderElection@818] - New election. My id =  5, proposed zxid=0x0
> 2016-07-21 08:11:46,030 [myid:] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,034 [myid:] - INFO  [/127.0.0.1:12228:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:38947
> 2016-07-21 08:11:46,034 [myid:] - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,034 [myid:] - INFO  [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,033 [myid:] - INFO  [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,035 [myid:] - INFO  [/127.0.0.1:12231:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:52089
> 2016-07-21 08:11:46,035 [myid:] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,036 [myid:] - INFO  [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,036 [myid:] - INFO  [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,037 [myid:] - INFO  [/127.0.0.1:12227:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:43442
> 2016-07-21 08:11:46,036 [myid:] - INFO  [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,036 [myid:] - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,037 [myid:] - INFO  [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,037 [myid:] - INFO  [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,038 [myid:] - INFO  [/127.0.0.1:12231:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:52091
> 2016-07-21 08:11:46,037 [myid:] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,036 [myid:] - INFO  [WorkerSender[myid=1]:QuorumCnxManager@199] - Have smaller server identifier, so dropping the connection: (5, 1)
> 2016-07-21 08:11:46,038 [myid:] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,038 [myid:] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,038 [myid:] - INFO  [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,038 [myid:] - INFO  [/127.0.0.1:12229:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:51573
> 2016-07-21 08:11:46,038 [myid:] - INFO  [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,037 [myid:] - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,039 [myid:] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,039 [myid:] - INFO  [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,039 [myid:] - INFO  [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,040 [myid:] - INFO  [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,039 [myid:] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,039 [myid:] - INFO  [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,039 [myid:] - INFO  [/127.0.0.1:12231:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:52093
> 2016-07-21 08:11:46,040 [myid:] - INFO  [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,040 [myid:] - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,040 [myid:] - INFO  [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,040 [myid:] - INFO  [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,041 [myid:] - INFO  [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,041 [myid:] - INFO  [/127.0.0.1:12227:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:43444
> 2016-07-21 08:11:46,041 [myid:] - INFO  [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,041 [myid:] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id = 5, error = 
> java.net.SocketException: Socket closed
> 	at java.net.SocketInputStream.socketRead0(Native Method)
> 	at java.net.SocketInputStream.read(SocketInputStream.java:152)
> 	at java.net.SocketInputStream.read(SocketInputStream.java:122)
> 	at java.io.DataInputStream.readFully(DataInputStream.java:195)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:805)
> 2016-07-21 08:11:46,042 [myid:] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:11:46,041 [myid:] - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,041 [myid:] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> 	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:11:46,041 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11222:NIOServerCnxn@827] - Processing stat command from /127.0.0.1:52363
> 2016-07-21 08:11:46,041 [myid:] - WARN  [RecvWorker:5:QuorumCnxManager$RecvWorker@810] - Connection broken for id 5, my id = 1, error = 
> java.io.EOFException
> 	at java.io.DataInputStream.readInt(DataInputStream.java:392)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:11:46,043 [myid:] - WARN  [RecvWorker:5:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:11:46,041 [myid:] - INFO  [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,041 [myid:] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,043 [myid:] - INFO  [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,043 [myid:] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:11:46,042 [myid:] - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,042 [myid:] - WARN  [SendWorker:5:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> 	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:11:46,044 [myid:] - WARN  [SendWorker:5:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:11:46,042 [myid:] - INFO  [WorkerReceiver[myid=4]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,042 [myid:] - INFO  [/127.0.0.1:12231:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:52096
> 2016-07-21 08:11:46,041 [myid:] - INFO  [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,045 [myid:] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> 	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:11:46,045 [myid:] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:11:46,045 [myid:] - INFO  [Thread-2:NIOServerCnxn@1008] - Closed socket connection for client /127.0.0.1:52363 (no session established for client)
> 2016-07-21 08:11:46,044 [myid:] - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,044 [myid:] - INFO  [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,044 [myid:] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,046 [myid:] - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,045 [myid:] - INFO  [/127.0.0.1:12227:QuorumCnxManager$Listener@541] - Received connection request /127.0.0.1:43446
> 2016-07-21 08:11:46,045 [myid:] - INFO  [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,045 [myid:] - WARN  [RecvWorker:5:QuorumCnxManager$RecvWorker@810] - Connection broken for id 5, my id = 1, error = 
> java.io.EOFException
> 	at java.io.DataInputStream.readInt(DataInputStream.java:392)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:11:46,047 [myid:] - WARN  [RecvWorker:5:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:11:46,045 [myid:] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id = 5, error = 
> java.net.SocketException: Socket closed
> 	at java.net.SocketInputStream.socketRead0(Native Method)
> 	at java.net.SocketInputStream.read(SocketInputStream.java:152)
> 	at java.net.SocketInputStream.read(SocketInputStream.java:122)
> 	at java.net.SocketInputStream.read(SocketInputStream.java:210)
> 	at java.io.DataInputStream.readInt(DataInputStream.java:387)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:11:46,047 [myid:] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:11:46,047 [myid:] - INFO  [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 4 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,047 [myid:] - WARN  [SendWorker:5:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> 	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:11:46,048 [myid:] - WARN  [SendWorker:5:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:11:46,046 [myid:] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,048 [myid:] - INFO  [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,048 [myid:] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,049 [myid:] - INFO  [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,049 [myid:] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 5 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,049 [myid:] - INFO  [WorkerReceiver[myid=5]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEpoch) LOOKING (my state)
> 2016-07-21 08:11:46,245 [myid:] - INFO  [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:QuorumPeer@844] - FOLLOWING
> 2016-07-21 08:11:46,246 [myid:] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11223:QuorumPeer@844] - FOLLOWING
> 2016-07-21 08:11:46,247 [myid:] - INFO  [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11224:QuorumPeer@844] - FOLLOWING
> 2016-07-21 08:11:46,249 [myid:] - INFO  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11222:QuorumPeer@844] - FOLLOWING
> 2016-07-21 08:11:46,249 [myid:] - INFO  [QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11226:QuorumPeer@856] - LEADING
> 2016-07-21 08:11:46,251 [myid:] - INFO  [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Learner@86] - TCP NoDelay set to: true
> 2016-07-21 08:11:46,255 [myid:] - INFO  [QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11226:Leader@59] - TCP NoDelay set to: true
> 2016-07-21 08:11:46,259 [myid:] - INFO  [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:zookeeper.version=3.4.9-SNAPSHOT-1753645, built on 07/21/2016 07:46 GMT
> 2016-07-21 08:11:46,259 [myid:] - INFO  [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:host.name=asf907.gq1.ygridcore.net
> 2016-07-21 08:11:46,260 [myid:] - INFO  [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:java.version=1.7.0_80
> 2016-07-21 08:11:46,260 [myid:] - INFO  [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:java.vendor=Oracle Corporation
> 2016-07-21 08:11:46,260 [myid:] - INFO  [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:java.home=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.7/jre
> 2016-07-21 08:11:46,260 [myid:] - INFO  [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:java.class.path=/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/classes:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/antlr-2.7.6.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/checkstyle-5.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-beanutils-core-1.7.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-cli-1.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-collections-3.2.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-lang-1.0.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/commons-logging-1.0.3.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/google-collections-0.9.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/junit-4.8.1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/test/lib/mockito-all-1.8.2.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/classes:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/src/java/lib/ivy-2.4.0.jar:/home/jenkins/tools/ant/latest/lib/ant.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/jline-0.9.94.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/log4j-1.2.16.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/netty-3.10.5.Final.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/slf4j-api-1.6.1.jar:/home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_jdk7/branch-3.4/build/lib/slf4j-log4j12-1.6.1.jar:/home/jenkins/tools/ant/apache-ant-1.9.4/lib/ant-launcher.jar:/home/jenkins/tools/ant/latest/lib/ant-junit.jar:/home/jenkins/tools/ant/latest/lib/ant-junit4.jar
> 2016-07-21 08:11:46,260 [myid:] - INFO  [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> 2016-07-21 08:11:46,260 [myid:] - INFO  [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:java.io.tmpdir=/tmp
> 2016-07-21 08:11:46,260 [myid:] - INFO  [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:java.compiler=<NA>
> 2016-07-21 08:11:46,260 [myid:] - INFO  [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:os.name=Linux
> 2016-07-21 08:11:46,260 [myid:] - INFO  [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:os.arch=amd64
> 2016-07-21 08:11:46,260 [myid:] - INFO  [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11225:Environment@100] - Server environment:os.version=3.13.0-36-lowlatency
> 2016-07-21 08:11:46,260 [myid:] - INFO  [QuorumPeer[myid=4]/0:0:0:
> ...[truncated 1420555 chars]...
> ager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:21,678 [myid:] - WARN  [SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:21,672 [myid:] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@810] - Connection broken for id 1, my id = 5, error = 
> java.io.EOFException
> 	at java.io.DataInputStream.readInt(DataInputStream.java:392)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:14:21,679 [myid:] - WARN  [RecvWorker:1:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:14:21,672 [myid:] - WARN  [SendWorker:5:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> 	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:21,679 [myid:] - WARN  [SendWorker:5:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:21,679 [myid:] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> 	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:21,679 [myid:] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:21,678 [myid:] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> 	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:21,679 [myid:] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:21,677 [myid:] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> 	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:21,680 [myid:] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:21,675 [myid:] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> 	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:21,680 [myid:] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:22,000 [myid:] - INFO  [SessionTracker:SessionTrackerImpl@162] - SessionTrackerImpl exited loop!
> 2016-07-21 08:14:22,317 [myid:] - INFO  [/127.0.0.1:12241:QuorumCnxManager$Listener@560] - Leaving listener
> 2016-07-21 08:14:22,320 [myid:] - INFO  [/127.0.0.1:12242:QuorumCnxManager$Listener@560] - Leaving listener
> 2016-07-21 08:14:22,579 [myid:] - INFO  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11233:Follower@166] - shutdown called
> java.lang.Exception: shutdown Follower
> 	at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
> 	at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:850)
> 2016-07-21 08:14:22,579 [myid:] - INFO  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11233:FollowerZooKeeperServer@140] - Shutting down
> 2016-07-21 08:14:22,579 [myid:] - INFO  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11233:SyncRequestProcessor@209] - Shutting down
> 2016-07-21 08:14:22,579 [myid:] - WARN  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:11233:QuorumPeer@874] - QuorumPeer main thread exited
> 2016-07-21 08:14:22,582 [myid:] - INFO  [main:QuorumBase@306] - Shutting down quorum peer QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234
> 2016-07-21 08:14:22,582 [myid:] - INFO  [main:Follower@166] - shutdown called
> java.lang.Exception: shutdown Follower
> 	at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
> 	at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:891)
> 	at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:307)
> 	at org.apache.zookeeper.test.QuorumBase.shutdownServers(QuorumBase.java:298)
> 	at org.apache.zookeeper.test.QuorumBase.tearDown(QuorumBase.java:285)
> 	at org.apache.zookeeper.test.QuorumZxidSyncTest.tearDown(QuorumZxidSyncTest.java:169)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> 	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> 	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> 	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> 	at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
> 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
> 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
> 	at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> 	at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
> 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:532)
> 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1179)
> 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1030)
> 2016-07-21 08:14:22,582 [myid:] - INFO  [main:FollowerZooKeeperServer@140] - Shutting down
> 2016-07-21 08:14:22,582 [myid:] - INFO  [main:ZooKeeperServer@469] - shutting down
> 2016-07-21 08:14:22,582 [myid:] - INFO  [main:FollowerRequestProcessor@107] - Shutting down
> 2016-07-21 08:14:22,582 [myid:] - INFO  [main:CommitProcessor@184] - Shutting down
> 2016-07-21 08:14:22,582 [myid:] - INFO  [FollowerRequestProcessor:2:FollowerRequestProcessor@97] - FollowerRequestProcessor exited loop!
> 2016-07-21 08:14:22,582 [myid:] - INFO  [CommitProcessor:2:CommitProcessor@153] - CommitProcessor exited loop!
> 2016-07-21 08:14:22,582 [myid:] - INFO  [main:FinalRequestProcessor@402] - shutdown of request processor complete
> 2016-07-21 08:14:22,583 [myid:] - INFO  [main:SyncRequestProcessor@209] - Shutting down
> 2016-07-21 08:14:22,583 [myid:] - INFO  [SyncThread:2:SyncRequestProcessor@187] - SyncRequestProcessor exited!
> 2016-07-21 08:14:22,584 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11234:NIOServerCnxnFactory@219] - NIOServerCnxn factory exited run method
> 2016-07-21 08:14:22,585 [myid:] - WARN  [SendWorker:3:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> 	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:22,585 [myid:] - WARN  [RecvWorker:2:QuorumCnxManager$RecvWorker@810] - Connection broken for id 2, my id = 4, error = 
> java.io.EOFException
> 	at java.io.DataInputStream.readInt(DataInputStream.java:392)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:14:22,585 [myid:] - WARN  [RecvWorker:2:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:14:22,585 [myid:] - WARN  [RecvWorker:3:QuorumCnxManager$RecvWorker@810] - Connection broken for id 3, my id = 2, error = 
> java.net.SocketException: Socket closed
> 	at java.net.SocketInputStream.socketRead0(Native Method)
> 	at java.net.SocketInputStream.read(SocketInputStream.java:152)
> 	at java.net.SocketInputStream.read(SocketInputStream.java:122)
> 	at java.net.SocketInputStream.read(SocketInputStream.java:210)
> 	at java.io.DataInputStream.readInt(DataInputStream.java:387)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:14:22,586 [myid:] - WARN  [RecvWorker:3:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:14:22,585 [myid:] - WARN  [RecvWorker:2:QuorumCnxManager$RecvWorker@810] - Connection broken for id 2, my id = 5, error = 
> java.io.EOFException
> 	at java.io.DataInputStream.readInt(DataInputStream.java:392)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:14:22,586 [myid:] - WARN  [RecvWorker:2:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:14:22,585 [myid:] - WARN  [RecvWorker:5:QuorumCnxManager$RecvWorker@810] - Connection broken for id 5, my id = 2, error = 
> java.net.SocketException: Socket closed
> 	at java.net.SocketInputStream.socketRead0(Native Method)
> 	at java.net.SocketInputStream.read(SocketInputStream.java:152)
> 	at java.net.SocketInputStream.read(SocketInputStream.java:122)
> 	at java.net.SocketInputStream.read(SocketInputStream.java:210)
> 	at java.io.DataInputStream.readInt(DataInputStream.java:387)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:14:22,587 [myid:] - WARN  [RecvWorker:5:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:14:22,585 [myid:] - WARN  [SendWorker:3:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:22,585 [myid:] - WARN  [RecvWorker:2:QuorumCnxManager$RecvWorker@810] - Connection broken for id 2, my id = 3, error = 
> java.io.EOFException
> 	at java.io.DataInputStream.readInt(DataInputStream.java:392)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:14:22,587 [myid:] - WARN  [RecvWorker:2:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:14:22,585 [myid:] - WARN  [SendWorker:5:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> 	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:22,587 [myid:] - WARN  [SendWorker:5:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:22,585 [myid:] - ERROR [/127.0.0.1:12239:QuorumCnxManager$Listener@547] - Exception while listening
> java.net.SocketException: Socket closed
> 	at java.net.PlainSocketImpl.socketAccept(Native Method)
> 	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
> 	at java.net.ServerSocket.implAccept(ServerSocket.java:530)
> 	at java.net.ServerSocket.accept(ServerSocket.java:498)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:539)
> 2016-07-21 08:14:22,587 [myid:] - WARN  [SendWorker:2:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> 	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:22,588 [myid:] - WARN  [SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:22,587 [myid:] - WARN  [SendWorker:2:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> 	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:22,589 [myid:] - WARN  [SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:22,586 [myid:] - WARN  [SendWorker:4:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> 	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:22,589 [myid:] - WARN  [SendWorker:4:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:22,586 [myid:] - WARN  [RecvWorker:4:QuorumCnxManager$RecvWorker@810] - Connection broken for id 4, my id = 2, error = 
> java.net.SocketException: Socket closed
> 	at java.net.SocketInputStream.socketRead0(Native Method)
> 	at java.net.SocketInputStream.read(SocketInputStream.java:152)
> 	at java.net.SocketInputStream.read(SocketInputStream.java:122)
> 	at java.net.SocketInputStream.read(SocketInputStream.java:210)
> 	at java.io.DataInputStream.readInt(DataInputStream.java:387)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:14:22,589 [myid:] - WARN  [RecvWorker:4:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:14:22,586 [myid:] - INFO  [main:QuorumBase@310] - Shutting down leader election QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234
> 2016-07-21 08:14:22,589 [myid:] - INFO  [main:QuorumBase@315] - Waiting for QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234 to exit thread
> 2016-07-21 08:14:22,585 [myid:] - WARN  [SendWorker:2:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> 	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:22,590 [myid:] - WARN  [SendWorker:2:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:22,673 [myid:] - INFO  [/127.0.0.1:12238:QuorumCnxManager$Listener@560] - Leaving listener
> 2016-07-21 08:14:23,079 [myid:] - INFO  [WorkerSender[myid=3]:FastLeaderElection$Messenger$WorkerSender@438] - WorkerSender is down
> 2016-07-21 08:14:23,080 [myid:] - INFO  [WorkerSender[myid=4]:FastLeaderElection$Messenger$WorkerSender@438] - WorkerSender is down
> 2016-07-21 08:14:23,084 [myid:] - INFO  [WorkerSender[myid=5]:FastLeaderElection$Messenger$WorkerSender@438] - WorkerSender is down
> 2016-07-21 08:14:23,088 [myid:] - INFO  [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@407] - WorkerReceiver is down
> 2016-07-21 08:14:23,088 [myid:] - INFO  [WorkerReceiver[myid=4]:FastLeaderElection$Messenger$WorkerReceiver@407] - WorkerReceiver is down
> 2016-07-21 08:14:23,089 [myid:] - INFO  [WorkerReceiver[myid=5]:FastLeaderElection$Messenger$WorkerReceiver@407] - WorkerReceiver is down
> 2016-07-21 08:14:23,578 [myid:] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234:Follower@166] - shutdown called
> java.lang.Exception: shutdown Follower
> 	at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
> 	at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:850)
> 2016-07-21 08:14:23,578 [myid:] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234:FollowerZooKeeperServer@140] - Shutting down
> 2016-07-21 08:14:23,578 [myid:] - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234:SyncRequestProcessor@209] - Shutting down
> 2016-07-21 08:14:23,578 [myid:] - WARN  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:11234:QuorumPeer@874] - QuorumPeer main thread exited
> 2016-07-21 08:14:23,579 [myid:] - INFO  [main:QuorumBase@306] - Shutting down quorum peer QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235
> 2016-07-21 08:14:23,579 [myid:] - INFO  [main:Follower@166] - shutdown called
> java.lang.Exception: shutdown Follower
> 	at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
> 	at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:891)
> 	at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:307)
> 	at org.apache.zookeeper.test.QuorumBase.shutdownServers(QuorumBase.java:299)
> 	at org.apache.zookeeper.test.QuorumBase.tearDown(QuorumBase.java:285)
> 	at org.apache.zookeeper.test.QuorumZxidSyncTest.tearDown(QuorumZxidSyncTest.java:169)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> 	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> 	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> 	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> 	at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
> 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
> 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
> 	at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> 	at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
> 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:532)
> 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1179)
> 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1030)
> 2016-07-21 08:14:23,579 [myid:] - INFO  [main:FollowerZooKeeperServer@140] - Shutting down
> 2016-07-21 08:14:23,579 [myid:] - INFO  [main:ZooKeeperServer@469] - shutting down
> 2016-07-21 08:14:23,580 [myid:] - INFO  [main:FollowerRequestProcessor@107] - Shutting down
> 2016-07-21 08:14:23,580 [myid:] - INFO  [main:CommitProcessor@184] - Shutting down
> 2016-07-21 08:14:23,580 [myid:] - INFO  [FollowerRequestProcessor:3:FollowerRequestProcessor@97] - FollowerRequestProcessor exited loop!
> 2016-07-21 08:14:23,580 [myid:] - INFO  [CommitProcessor:3:CommitProcessor@153] - CommitProcessor exited loop!
> 2016-07-21 08:14:23,580 [myid:] - INFO  [main:FinalRequestProcessor@402] - shutdown of request processor complete
> 2016-07-21 08:14:23,581 [myid:] - INFO  [main:SyncRequestProcessor@209] - Shutting down
> 2016-07-21 08:14:23,581 [myid:] - INFO  [SyncThread:3:SyncRequestProcessor@187] - SyncRequestProcessor exited!
> 2016-07-21 08:14:23,582 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11235:NIOServerCnxnFactory@219] - NIOServerCnxn factory exited run method
> 2016-07-21 08:14:23,583 [myid:] - ERROR [/127.0.0.1:12240:QuorumCnxManager$Listener@547] - Exception while listening
> java.net.SocketException: Socket closed
> 	at java.net.PlainSocketImpl.socketAccept(Native Method)
> 	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
> 	at java.net.ServerSocket.implAccept(ServerSocket.java:530)
> 	at java.net.ServerSocket.accept(ServerSocket.java:498)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:539)
> 2016-07-21 08:14:23,583 [myid:] - WARN  [SendWorker:5:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> 	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:23,585 [myid:] - WARN  [SendWorker:5:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:23,584 [myid:] - WARN  [RecvWorker:4:QuorumCnxManager$RecvWorker@810] - Connection broken for id 4, my id = 3, error = 
> java.net.SocketException: Socket closed
> 	at java.net.SocketInputStream.socketRead0(Native Method)
> 	at java.net.SocketInputStream.read(SocketInputStream.java:152)
> 	at java.net.SocketInputStream.read(SocketInputStream.java:122)
> 	at java.net.SocketInputStream.read(SocketInputStream.java:210)
> 	at java.io.DataInputStream.readInt(DataInputStream.java:387)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:14:23,585 [myid:] - WARN  [RecvWorker:4:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:14:23,584 [myid:] - WARN  [SendWorker:4:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> 	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:23,585 [myid:] - WARN  [SendWorker:4:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:23,584 [myid:] - WARN  [RecvWorker:5:QuorumCnxManager$RecvWorker@810] - Connection broken for id 5, my id = 3, error = 
> java.net.SocketException: Socket closed
> 	at java.net.SocketInputStream.socketRead0(Native Method)
> 	at java.net.SocketInputStream.read(SocketInputStream.java:152)
> 	at java.net.SocketInputStream.read(SocketInputStream.java:122)
> 	at java.net.SocketInputStream.read(SocketInputStream.java:210)
> 	at java.io.DataInputStream.readInt(DataInputStream.java:387)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:14:23,585 [myid:] - WARN  [RecvWorker:5:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:14:23,584 [myid:] - WARN  [RecvWorker:3:QuorumCnxManager$RecvWorker@810] - Connection broken for id 3, my id = 5, error = 
> java.io.EOFException
> 	at java.io.DataInputStream.readInt(DataInputStream.java:392)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:14:23,586 [myid:] - WARN  [RecvWorker:3:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:14:23,584 [myid:] - WARN  [RecvWorker:3:QuorumCnxManager$RecvWorker@810] - Connection broken for id 3, my id = 4, error = 
> java.io.EOFException
> 	at java.io.DataInputStream.readInt(DataInputStream.java:392)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:14:23,586 [myid:] - WARN  [RecvWorker:3:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:14:23,584 [myid:] - INFO  [main:QuorumBase@310] - Shutting down leader election QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235
> 2016-07-21 08:14:23,586 [myid:] - INFO  [main:QuorumBase@315] - Waiting for QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235 to exit thread
> 2016-07-21 08:14:23,586 [myid:] - WARN  [SendWorker:3:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> 	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:23,586 [myid:] - WARN  [SendWorker:3:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:23,586 [myid:] - WARN  [SendWorker:3:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> 	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:23,587 [myid:] - WARN  [SendWorker:3:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:23,588 [myid:] - INFO  [/127.0.0.1:12239:QuorumCnxManager$Listener@560] - Leaving listener
> 2016-07-21 08:14:24,355 [myid:] - INFO  [WorkerSender[myid=2]:FastLeaderElection$Messenger$WorkerSender@438] - WorkerSender is down
> 2016-07-21 08:14:24,358 [myid:] - INFO  [WorkerSender[myid=1]:FastLeaderElection$Messenger$WorkerSender@438] - WorkerSender is down
> 2016-07-21 08:14:24,360 [myid:] - INFO  [WorkerSender[myid=3]:FastLeaderElection$Messenger$WorkerSender@438] - WorkerSender is down
> 2016-07-21 08:14:24,361 [myid:] - INFO  [WorkerReceiver[myid=1]:FastLeaderElection$Messenger$WorkerReceiver@407] - WorkerReceiver is down
> 2016-07-21 08:14:24,361 [myid:] - INFO  [WorkerReceiver[myid=2]:FastLeaderElection$Messenger$WorkerReceiver@407] - WorkerReceiver is down
> 2016-07-21 08:14:24,362 [myid:] - INFO  [WorkerReceiver[myid=3]:FastLeaderElection$Messenger$WorkerReceiver@407] - WorkerReceiver is down
> 2016-07-21 08:14:24,578 [myid:] - INFO  [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235:Follower@166] - shutdown called
> java.lang.Exception: shutdown Follower
> 	at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
> 	at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:850)
> 2016-07-21 08:14:24,578 [myid:] - INFO  [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235:FollowerZooKeeperServer@140] - Shutting down
> 2016-07-21 08:14:24,578 [myid:] - INFO  [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235:SyncRequestProcessor@209] - Shutting down
> 2016-07-21 08:14:24,578 [myid:] - WARN  [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:11235:QuorumPeer@874] - QuorumPeer main thread exited
> 2016-07-21 08:14:24,579 [myid:] - INFO  [main:QuorumBase@306] - Shutting down quorum peer QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236
> 2016-07-21 08:14:24,579 [myid:] - INFO  [main:Follower@166] - shutdown called
> java.lang.Exception: shutdown Follower
> 	at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
> 	at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:891)
> 	at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:307)
> 	at org.apache.zookeeper.test.QuorumBase.shutdownServers(QuorumBase.java:300)
> 	at org.apache.zookeeper.test.QuorumBase.tearDown(QuorumBase.java:285)
> 	at org.apache.zookeeper.test.QuorumZxidSyncTest.tearDown(QuorumZxidSyncTest.java:169)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> 	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> 	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> 	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> 	at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
> 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
> 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
> 	at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> 	at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
> 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:532)
> 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1179)
> 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1030)
> 2016-07-21 08:14:24,580 [myid:] - INFO  [main:FollowerZooKeeperServer@140] - Shutting down
> 2016-07-21 08:14:24,581 [myid:] - INFO  [main:ZooKeeperServer@469] - shutting down
> 2016-07-21 08:14:24,581 [myid:] - INFO  [main:FollowerRequestProcessor@107] - Shutting down
> 2016-07-21 08:14:24,581 [myid:] - INFO  [main:CommitProcessor@184] - Shutting down
> 2016-07-21 08:14:24,581 [myid:] - INFO  [main:FinalRequestProcessor@402] - shutdown of request processor complete
> 2016-07-21 08:14:24,581 [myid:] - INFO  [FollowerRequestProcessor:4:FollowerRequestProcessor@97] - FollowerRequestProcessor exited loop!
> 2016-07-21 08:14:24,581 [myid:] - INFO  [main:SyncRequestProcessor@209] - Shutting down
> 2016-07-21 08:14:24,581 [myid:] - INFO  [CommitProcessor:4:CommitProcessor@153] - CommitProcessor exited loop!
> 2016-07-21 08:14:24,582 [myid:] - INFO  [SyncThread:4:SyncRequestProcessor@187] - SyncRequestProcessor exited!
> 2016-07-21 08:14:24,583 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11236:NIOServerCnxnFactory@219] - NIOServerCnxn factory exited run method
> 2016-07-21 08:14:24,585 [myid:] - INFO  [main:QuorumBase@310] - Shutting down leader election QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236
> 2016-07-21 08:14:24,586 [myid:] - ERROR [/127.0.0.1:12241:QuorumCnxManager$Listener@547] - Exception while listening
> java.net.SocketException: Socket closed
> 	at java.net.PlainSocketImpl.socketAccept(Native Method)
> 	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
> 	at java.net.ServerSocket.implAccept(ServerSocket.java:530)
> 	at java.net.ServerSocket.accept(ServerSocket.java:498)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:539)
> 2016-07-21 08:14:24,586 [myid:] - WARN  [SendWorker:5:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> 	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:24,586 [myid:] - WARN  [SendWorker:5:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:24,586 [myid:] - INFO  [main:QuorumBase@315] - Waiting for QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236 to exit thread
> 2016-07-21 08:14:24,586 [myid:] - WARN  [RecvWorker:4:QuorumCnxManager$RecvWorker@810] - Connection broken for id 4, my id = 5, error = 
> java.io.EOFException
> 	at java.io.DataInputStream.readInt(DataInputStream.java:392)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:14:24,587 [myid:] - WARN  [RecvWorker:4:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:14:24,585 [myid:] - INFO  [/127.0.0.1:12240:QuorumCnxManager$Listener@560] - Leaving listener
> 2016-07-21 08:14:24,585 [myid:] - WARN  [RecvWorker:5:QuorumCnxManager$RecvWorker@810] - Connection broken for id 5, my id = 4, error = 
> java.net.SocketException: Socket closed
> 	at java.net.SocketInputStream.socketRead0(Native Method)
> 	at java.net.SocketInputStream.read(SocketInputStream.java:152)
> 	at java.net.SocketInputStream.read(SocketInputStream.java:122)
> 	at java.net.SocketInputStream.read(SocketInputStream.java:210)
> 	at java.io.DataInputStream.readInt(DataInputStream.java:387)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:795)
> 2016-07-21 08:14:24,588 [myid:] - WARN  [RecvWorker:5:QuorumCnxManager$RecvWorker@813] - Interrupting SendWorker
> 2016-07-21 08:14:24,588 [myid:] - WARN  [SendWorker:4:QuorumCnxManager$SendWorker@727] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2095)
> 	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:389)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:879)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:65)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:715)
> 2016-07-21 08:14:24,588 [myid:] - WARN  [SendWorker:4:QuorumCnxManager$SendWorker@736] - Send worker leaving thread
> 2016-07-21 08:14:25,579 [myid:] - INFO  [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236:Follower@166] - shutdown called
> java.lang.Exception: shutdown Follower
> 	at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
> 	at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:850)
> 2016-07-21 08:14:25,579 [myid:] - INFO  [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236:FollowerZooKeeperServer@140] - Shutting down
> 2016-07-21 08:14:25,579 [myid:] - INFO  [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236:SyncRequestProcessor@209] - Shutting down
> 2016-07-21 08:14:25,579 [myid:] - WARN  [QuorumPeer[myid=4]/0:0:0:0:0:0:0:0:11236:QuorumPeer@874] - QuorumPeer main thread exited
> 2016-07-21 08:14:25,579 [myid:] - INFO  [main:QuorumBase@306] - Shutting down quorum peer QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11237
> 2016-07-21 08:14:25,579 [myid:] - INFO  [main:Leader@496] - Shutting down
> 2016-07-21 08:14:25,580 [myid:] - INFO  [main:Leader@502] - Shutdown called
> java.lang.Exception: shutdown Leader! reason: quorum Peer shutdown
> 	at org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:502)
> 	at org.apache.zookeeper.server.quorum.QuorumPeer.shutdown(QuorumPeer.java:888)
> 	at org.apache.zookeeper.test.QuorumBase.shutdown(QuorumBase.java:307)
> 	at org.apache.zookeeper.test.QuorumBase.shutdownServers(QuorumBase.java:301)
> 	at org.apache.zookeeper.test.QuorumBase.tearDown(QuorumBase.java:285)
> 	at org.apache.zookeeper.test.QuorumZxidSyncTest.tearDown(QuorumZxidSyncTest.java:169)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> 	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> 	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> 	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> 	at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:48)
> 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
> 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
> 	at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> 	at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
> 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:532)
> 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1179)
> 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1030)
> 2016-07-21 08:14:25,581 [myid:] - INFO  [main:ZooKeeperServer@469] - shutting down
> 2016-07-21 08:14:25,581 [myid:] - INFO  [LearnerCnxAcceptor-/127.0.0.1:12237:Leader$LearnerCnxAcceptor@325] - exception while shutting down acceptor: java.net.SocketException: Socket closed
> 2016-07-21 08:14:25,582 [myid:] - INFO  [main:SessionTrackerImpl@225] - Shutting down
> 2016-07-21 08:14:25,582 [myid:] - INFO  [main:PrepRequestProcessor@765] - Shutting down
> 2016-07-21 08:14:25,582 [myid:] - INFO  [main:ProposalRequestProcessor@88] - Shutting down
> 2016-07-21 08:14:25,582 [myid:] - INFO  [main:CommitProcessor@184] - Shutting down
> 2016-07-21 08:14:25,582 [myid:] - INFO  [ProcessThread(sid:5 cport:-1)::PrepRequestProcessor@143] - PrepRequestProcessor exited loop!
> 2016-07-21 08:14:25,583 [myid:] - INFO  [CommitProcessor:5:CommitProcessor@153] - CommitProcessor exited loop!
> 2016-07-21 08:14:25,582 [myid:] - INFO  [main:Leader$ToBeAppliedRequestProcessor@661] - Shutting down
> 2016-07-21 08:14:25,583 [myid:] - INFO  [main:FinalRequestProcessor@402] - shutdown of request processor complete
> 2016-07-21 08:14:25,583 [myid:] - INFO  [main:SyncRequestProcessor@209] - Shutting down
> 2016-07-21 08:14:25,583 [myid:] - INFO  [SyncThread:5:SyncRequestProcessor@187] - SyncRequestProcessor exited!
> 2016-07-21 08:14:25,585 [myid:] - WARN  [LearnerHandler-/127.0.0.1:43485:LearnerHandler@644] - ******* GOODBYE /127.0.0.1:43485 ********
> 2016-07-21 08:14:25,585 [myid:] - WARN  [LearnerHandler-/127.0.0.1:43486:LearnerHandler@644] - ******* GOODBYE /127.0.0.1:43486 ********
> 2016-07-21 08:14:25,585 [myid:] - WARN  [LearnerHandler-/127.0.0.1:43486:LearnerHandler@656] - Ignoring unexpected exception
> java.lang.InterruptedException
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
> 	at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
> 	at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
> 	at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:654)
> 	at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:647)
> 2016-07-21 08:14:25,585 [myid:] - WARN  [LearnerHandler-/127.0.0.1:43485:LearnerHandler@656] - Ignoring unexpected exception
> java.lang.InterruptedException
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
> 	at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
> 	at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
> 	at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:654)
> 	at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:647)
> 2016-07-21 08:14:25,585 [myid:] - WARN  [LearnerHandler-/127.0.0.1:43488:LearnerHandler@644] - ******* GOODBYE /127.0.0.1:43488 ********
> 2016-07-21 08:14:25,586 [myid:] - INFO  [/127.0.0.1:12241:QuorumCnxManager$Listener@560] - Leaving listener
> 2016-07-21 08:14:25,586 [myid:] - WARN  [LearnerHandler-/127.0.0.1:43487:LearnerHandler@644] - ******* GOODBYE /127.0.0.1:43487 ********
> 2016-07-21 08:14:25,586 [myid:] - WARN  [LearnerHandler-/127.0.0.1:43487:LearnerHandler@656] - Ignoring unexpected exception
> java.lang.InterruptedException
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
> 	at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
> 	at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
> 	at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:654)
> 	at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:647)
> 2016-07-21 08:14:25,586 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:11237:NIOServerCnxnFactory@219] - NIOServerCnxn factory exited run method
> 2016-07-21 08:14:25,586 [myid:] - WARN  [LearnerHandler-/127.0.0.1:43488:LearnerHandler@656] - Ignoring unexpected exception
> java.lang.InterruptedException
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
> 	at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
> 	at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
> 	at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:654)
> 	at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:647)
> 2016-07-21 08:14:25,587 [myid:] - WARN  [QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11237:QuorumPeer@862] - Unexpected exception
> java.lang.InterruptedException: sleep interrupted
> 	at java.lang.Thread.sleep(Native Method)
> 	at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:456)
> 	at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:859)
> 2016-07-21 08:14:25,588 [myid:] - INFO  [QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11237:Leader@496] - Shutting down
> 2016-07-21 08:14:25,588 [myid:] - INFO  [main:QuorumBase@310] - Shutting down leader election QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11237
> 2016-07-21 08:14:25,588 [myid:] - ERROR [/127.0.0.1:12242:QuorumCnxManager$Listener@547] - Exception while listening
> java.net.SocketException: Socket closed
> 	at java.net.PlainSocketImpl.socketAccept(Native Method)
> 	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
> 	at java.net.ServerSocket.implAccept(ServerSocket.java:530)
> 	at java.net.ServerSocket.accept(ServerSocket.java:498)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:539)
> 2016-07-21 08:14:25,588 [myid:] - INFO  [main:QuorumBase@315] - Waiting for QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11237 to exit thread
> 2016-07-21 08:14:25,588 [myid:] - WARN  [QuorumPeer[myid=5]/0:0:0:0:0:0:0:0:11237:QuorumPeer@874] - QuorumPeer main thread exited
> 2016-07-21 08:14:25,589 [myid:] - INFO  [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11233
> 2016-07-21 08:14:25,589 [myid:] - INFO  [main:QuorumBase@291] - 127.0.0.1:11233 is no longer accepting client connections
> 2016-07-21 08:14:25,589 [myid:] - INFO  [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11234
> 2016-07-21 08:14:25,589 [myid:] - INFO  [main:QuorumBase@291] - 127.0.0.1:11234 is no longer accepting client connections
> 2016-07-21 08:14:25,590 [myid:] - INFO  [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11235
> 2016-07-21 08:14:25,590 [myid:] - INFO  [main:QuorumBase@291] - 127.0.0.1:11235 is no longer accepting client connections
> 2016-07-21 08:14:25,590 [myid:] - INFO  [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11236
> 2016-07-21 08:14:25,590 [myid:] - INFO  [main:QuorumBase@291] - 127.0.0.1:11236 is no longer accepting client connections
> 2016-07-21 08:14:25,590 [myid:] - INFO  [main:FourLetterWordMain@62] - connecting to 127.0.0.1 11237
> 2016-07-21 08:14:25,590 [myid:] - INFO  [main:QuorumBase@291] - 127.0.0.1:11237 is no longer accepting client connections
> 2016-07-21 08:14:25,592 [myid:] - INFO  [main:ZKTestCase$1@60] - SUCCEEDED testLateLogs
> 2016-07-21 08:14:25,592 [myid:] - INFO  [main:ZKTestCase$1@55] - FINISHED testLateLogs
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message