flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Samir Tusharbhai Chauhan <samir.tusharbhai.chau...@prudential.com.sg>
Subject Timeout error in ZooKeeper
Date Fri, 28 Feb 2020 18:41:09 GMT

Yesterday morning I got below error in Zookeeper. After this error, my Flink did not connect
to ZK and jobs went to hang state. I had to cancel and redeploy my all jobs to bring it to
normal state.
2020-02-28 02:45:56,811 [myid:1] - WARN  [NIOServerCxn.Factory:]
- caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x1701028573403f3,
likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
        at java.lang.Thread.run(Thread.java:748)
At the same time I saw below error in Flink.
2020-02-28 02:46:49,095 ERROR org.apache.curator.ConnectionState                         
  - Connection timed out for connection string (zk-cs:2181) and timeout (3000) / elapsed (14305)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
      at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:225)
      at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:94)
      at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:117)
      at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835)
      at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809)
      at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64)
      at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)

Has anyone face similar error earlier.

My environment is
Azure Kubernetes 1.15.7
Flink 1.6.0
Zookeeper 3.4.10

Warm Regards,
Samir Chauhan

There's a reason we support Fair Dealing. YOU.

This email and any files transmitted with it or attached to it (the [Email]) may contain confidential,
proprietary or legally privileged information and is intended solely for the use of the individual
or entity to whom it is addressed. If you are not the intended recipient of the Email, you
must not, directly or indirectly, copy, use, print, distribute, disclose to any other party
or take any action in reliance on any part of the Email. Please notify the system manager
or sender of the error and delete all copies of the Email immediately.  

No statement in the Email should be construed as investment advice being given within or outside
Singapore. Prudential Assurance Company Singapore (Pte) Limited (PACS)  and each of its related
entities shall not be responsible for any losses, claims, penalties, costs or damages arising
from or in connection with the use of the Email or the information therein, in whole or in
part. You are solely responsible for conducting any virus checks prior to opening, accessing
or disseminating the Email.

PACS (Company Registration No. 199002477Z) is a company incorporated under the laws of Singapore
and has its registered office at 30 Cecil Street, #30-01, Prudential Tower, Singapore 049712.

PACS is an indirect wholly owned subsidiary of Prudential plc of the United Kingdom. PACS
and Prudential plc are not affiliated in any manner with Prudential Financial, Inc., a company
whose principal place of business is in the United States of America.
View raw message