It's also likely you have a fair bit of packet loss between your
datacenters, unless you know you have a solid network between them. If your
observers are falling offline "randomly", packet loss is a pretty likely
culprit.
On Thu, Mar 9, 2017 at 9:54 AM, Michael Han <hanm@cloudera.com> wrote:
> The log indicates that your server socket on observer timed out after
> syncing with leader. It could simply because that the latency between your
> DCs exceeds the socket timeout configuration ZK uses. The timeout is
> calculated as tickTime * syncLimit so you might want tweak these values to
> fit the latency between your DCs.
>
> On Thu, Mar 9, 2017 at 9:00 AM, rammohan ganapavarapu <
> rammohanganap@gmail.com> wrote:
>
> > Hi,
> >
> > We have a multi data-center zk cluster with all the followers are in one
> > data-center and observers in other data-centers, for some reason
> observers
> > are going down with the following exception and i am not sure what could
> be
> > the reason and how to avoid this issue, any thoughts?
> >
> > Ram
> >
> >
> >
> > 2017-03-09 09:00:18,305 - WARN
> > [QuorumPeer[myid=41]/0:0:0:0:0:0:0:0:2181:Observer@79] - Exception when
> > observing the leader
> > java.net.SocketTimeoutException: Read timed out
> > at java.net.SocketInputStream.socketRead0(Native Method)
> > at java.net.SocketInputStream.read(SocketInputStream.java:152)
> > at java.net.SocketInputStream.read(SocketInputStream.java:122)
> > at java.io.BufferedInputStream.fill(BufferedInputStream.java:
> 235)
> > at java.io.BufferedInputStream.read(BufferedInputStream.java:
> 254)
> > at java.io.DataInputStream.readInt(DataInputStream.java:387)
> > at
> > org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> > at
> > org.apache.zookeeper.server.quorum.QuorumPacket.
> > deserialize(QuorumPacket.java:83)
> > at
> > org.apache.jute.BinaryInputArchive.readRecord(
> BinaryInputArchive.java:108)
> > at
> > org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:152)
> > at
> > org.apache.zookeeper.server.quorum.Observer.observeLeader(
> > Observer.java:75)
> > at
> > org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:727)
> > 2017-03-09 09:00:18,306 - INFO
> > [QuorumPeer[myid=41]/0:0:0:0:0:0:0:0:2181:Observer@137] - shutdown
> called
> > java.lang.Exception: shutdown Observer
> > at
> > org.apache.zookeeper.server.quorum.Observer.shutdown(Observer.java:137)
> >
>
>
>
> --
> Cheers
> Michael.
>
|