zookeeper-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jx (Jira)" <j...@apache.org>
Subject [jira] [Created] (ZOOKEEPER-3673) Getting a snapshot from leader cause Connection reset shutdown Follower and repeated forever
Date Mon, 06 Jan 2020 02:46:00 GMT
jx created ZOOKEEPER-3673:
-----------------------------

             Summary: Getting a snapshot from leader cause Connection reset shutdown Follower
and repeated forever
                 Key: ZOOKEEPER-3673
                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3673
             Project: ZooKeeper
          Issue Type: Bug
    Affects Versions: 3.4.12
            Reporter: jx


when one broker restart, zk repeated forever

1. Getting a snapshot from leader

2. Snapshotting to disk

3. cause Connection reset

4. shutdown Follower

 

Does get snapshot from leader or snapshot to disk cause synclimit timeout ?
{code:java}
// code placeholder
2020-01-05 22:56:31,168 [myid:3] - INFO  [QuorumPeer[myid=3]/0.0.0.0:2181:Environment@100]
- Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2020-01-05 22:56:31,169 [myid:3] - INFO  [QuorumPeer[myid=3]/0.0.0.0:2181:Environment@100]
- Server environment:java.io.tmpdir=/tmp
2020-01-05 22:56:31,169 [myid:3] - INFO  [QuorumPeer[myid=3]/0.0.0.0:2181:Environment@100]
- Server environment:java.compiler=<NA>
2020-01-05 22:56:31,169 [myid:3] - INFO  [QuorumPeer[myid=3]/0.0.0.0:2181:Environment@100]
- Server environment:os.name=Linux
2020-01-05 22:56:31,169 [myid:3] - INFO  [QuorumPeer[myid=3]/0.0.0.0:2181:Environment@100]
- Server environment:os.arch=amd64
2020-01-05 22:56:31,169 [myid:3] - INFO  [QuorumPeer[myid=3]/0.0.0.0:2181:Environment@100]
- Server environment:os.version=3.10.104-1-tlinux2_kvm_guest-0022.tl2
2020-01-05 22:56:31,169 [myid:3] - INFO  [QuorumPeer[myid=3]/0.0.0.0:2181:Environment@100]
- Server environment:user.name=user_00
2020-01-05 22:56:31,169 [myid:3] - INFO  [QuorumPeer[myid=3]/0.0.0.0:2181:Environment@100]
- Server environment:user.home=/home/user_00
2020-01-05 22:56:31,170 [myid:3] - INFO  [QuorumPeer[myid=3]/0.0.0.0:2181:Environment@100]
- Server environment:user.dir=/usr/local/services/zookeeper-3_4_12-V8-32-400-cluster-001-0.0
2020-01-05 22:56:31,171 [myid:3] - INFO  [QuorumPeer[myid=3]/0.0.0.0:2181:ZooKeeperServer@173]
- Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir
/data/zookeeper/version-2 snapdir /data/zookeeper/version-2
2020-01-05 22:56:31,183 [myid:3] - INFO  [QuorumPeer[myid=3]/0.0.0.0:2181:Follower@65] - FOLLOWING
- LEADER ELECTION TOOK - 81
2020-01-05 22:56:31,185 [myid:3] - INFO  [QuorumPeer[myid=3]/0.0.0.0:2181:QuorumPeer$QuorumServer@184]
- Resolved hostname: 100.94.122.151 to address: /100.94.122.151
2020-01-05 22:56:31,190 [myid:3] - INFO  [QuorumPeer[myid=3]/0.0.0.0:2181:Learner@336] - Getting
a snapshot from leader 0xb1a0a15a6
2020-01-05 22:57:19,023 [myid:3] - INFO  [QuorumPeer[myid=3]/0.0.0.0:2181:FileTxnSnapLog@296]
- Snapshotting: 0xb1a0a15a6 to /data/zookeeper/version-2/snapshot.b1a0a15a6
2020-01-05 22:57:53,554 [myid:3] - WARN  [QuorumPeer[myid=3]/0.0.0.0:2181:Learner@387] - Got
zxid 0xb1a0a15a7 expected 0x1
2020-01-05 22:57:53,596 [myid:3] - WARN  [QuorumPeer[myid=3]/0.0.0.0:2181:Follower@90] - Exception
when following the leader
java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:210)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
        at java.io.DataInputStream.readFully(DataInputStream.java:195)
        at java.io.DataInputStream.readFully(DataInputStream.java:169)
        at org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:94)
        at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:87)
        at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
        at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:153)
        at org.apache.zookeeper.server.quorum.Learner.syncWithLeader(Learner.java:380)
        at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:83)
        at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:981)
2020-01-05 22:57:53,615 [myid:3] - INFO  [QuorumPeer[myid=3]/0.0.0.0:2181:Follower@169] -
shutdown called
java.lang.Exception: shutdown Follower
        at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:169)
        at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:985)
2020-01-05 22:57:53,615 [myid:3] - INFO  [QuorumPeer[myid=3]/0.0.0.0:2181:FollowerZooKeeperServer@140]
- Shutting down
2020-01-05 22:57:53,615 [myid:3] - INFO  [QuorumPeer[myid=3]/0.0.0.0:2181:QuorumPeer@909]
- LOOKING
2020-01-05 22:57:53,616 [myid:3] - INFO  [QuorumPeer[myid=3]/0.0.0.0:2181:FastLeaderElection@813]
- New election. My id =  3, proposed zxid=0xb1a0a15a6
2020-01-05 22:57:53,617 [myid:3] - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@595] -
Notification: 1 (message format version), 3 (n.leader), 0xb1a0a15a6 (n.zxid), 0x2 (n.round),
LOOKING (n.state), 3 (n.sid), 0xb (n.peerEpoch) LOOKING (my state)
2020-01-05 22:57:53,618 [myid:3] - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@595] -
Notification: 1 (message format version), 2 (n.leader), 0xa0000001b (n.zxid), 0x1 (n.round),
FOLLOWING (n.state), 1 (n.sid), 0xb (n.peerEpoch) LOOKING (my state)
2020-01-05 22:57:53,618 [myid:3] - INFO  [WorkerReceiver[myid=3]:FastLeaderElection@595] -
Notification: 1 (message format version), 2 (n.leader), 0xa0000001b (n.zxid), 0x1 (n.round),
LEADING (n.state), 2 (n.sid), 0xb (n.peerEpoch) LOOKING (my state)
2020-01-05 22:57:53,619 [myid:3] - INFO  [QuorumPeer[myid=3]/0.0.0.0:2181:QuorumPeer@979]
- FOLLOWING
2020-01-05 22:57:53,619 [myid:3] - INFO  [QuorumPeer[myid=3]/0.0.0.0:2181:ZooKeeperServer@173]
- Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir
/data/zookeeper/version-2 snapdir /data/zookeeper/version-2
2020-01-05 22:57:53,619 [myid:3] - INFO  [QuorumPeer[myid=3]/0.0.0.0:2181:Follower@65] - FOLLOWING
- LEADER ELECTION TOOK - 3
2020-01-05 22:57:53,619 [myid:3] - INFO  [QuorumPeer[myid=3]/0.0.0.0:2181:QuorumPeer$QuorumServer@184]
- Resolved hostname: 100.94.122.151 to address: /100.94.122.151
2020-01-05 22:57:53,628 [myid:3] - INFO  [QuorumPeer[myid=3]/0.0.0.0:2181:Learner@336] - Getting
a snapshot from leader 0xb1a0a4842
2020-01-05 22:58:34,196 [myid:3] - INFO  [QuorumPeer[myid=3]/0.0.0.0:2181:FileTxnSnapLog@296]
- Snapshotting: 0xb1a0a4842 to /data/zookeeper/version-2/snapshot.b1a0a4842
2020-01-05 22:59:03,670 [myid:3] - WARN  [QuorumPeer[myid=3]/0.0.0.0:2181:Learner@387] - Got
zxid 0xb1a0a4843 expected 0x1
2020-01-05 22:59:03,692 [myid:3] - WARN  [QuorumPeer[myid=3]/0.0.0.0:2181:Follower@90] - Exception
when following the leader
java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:210)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message