zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Norbert Kalmar <nkal...@cloudera.com.INVALID>
Subject Re: Zookeeper crashes with EOF Exception
Date Wed, 27 Feb 2019 09:04:43 GMT
Hi JS,

Looks like there was a Leader election, and during sync phase
(syncWithLeader), the follower tried to deserialize the snapshot, but it is
an incomplete file, hence the EOF exception.
How big is your snapshot? Did you run out of disc space?
Also worth checking for fsync warnings / errors in the log.

Hope this helps.

Regards,
Norbert

On Wed, Feb 27, 2019 at 8:05 AM zoo_js <sathish.js.kumar@cyber-itus.com>
wrote:

> Hi all,
>
> We have a 3 node zookeeper cluster used for Vault as HA.  Starting a few
> days ago, the entire cluster crashes a few times per day, all nodes at the
> exact same time. We are running some load test using vault for Data
> encryption. Per minute 1000 keys unique keys will be generated, Once the
> issue started around 270,000 keys.
>
> The following exception is got from the syslog, not sure what's causing
> this
> crash. Please help to proceed..
>
> 2019-02-26 22:35:18,831 [myid:1] - WARN
> [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Follower@90] - Exception when
> following the leader
> java.io.EOFException
>        at
> java.base/java.io.DataInputStream.readFully(DataInputStream.java:202)
>        at
> java.base/java.io.DataInputStream.readFully(DataInputStream.java:170)
>        at
> org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:94)
>        at
> org.apache.zookeeper.server.DataNode.deserialize(DataNode.java:165)
>        at
> org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
>        at
> org.apache.zookeeper.server.DataTree.deserialize(DataTree.java:1076)
>        at
>
> org.apache.zookeeper.server.util.SerializeUtils.deserializeSnapshot(SerializeUtils.java:130)
>        at
>
> org.apache.zookeeper.server.ZKDatabase.deserializeSnapshot(ZKDatabase.java:452)
>        at
> org.apache.zookeeper.server.quorum.Learner.syncWithLeader(Learner.java:340)
>        at
> org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:83)
>        at
> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:981)
> 2019-02-26 22:35:19,349 [myid:1] - INFO
> [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Follower@169] - shutdown called
> java.lang.Exception: shutdown Follower
>        at
> org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:169)
>
> thanks
> JS
>
>
>
>
>
>
> --
> Sent from: http://zookeeper-user.578899.n2.nabble.com/
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message