flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Renkai <gaelook...@gmail.com>
Subject JobManager shows TaskManager was lost/killed while TaskManger Process is still running and the network is OK.
Date Fri, 25 Nov 2016 03:54:50 GMT
I use the newest snapshot of Flink, all jobs failed since a TaskManager was lost/killed.There
is a sample of jobmanager and taskmanager logs

 

//job manager

java.lang.Exception: TaskManager was lost/killed: ResourceID{resourceId='8f4b98897b1cbdbb576cbf298ac1339f'}
@ 10.17.123.56 (dataPort=62636)

        at org.apache.flink.runtime.instance.SimpleSlot.releaseSlot(SimpleSlot.java:217)

        at org.apache.flink.runtime.instance.SlotSharingGroupAssignment.releaseSharedSlot(SlotSharingGroupAssignment.java:533)

        at org.apache.flink.runtime.instance.SharedSlot.releaseSlot(SharedSlot.java:192)

        at org.apache.flink.runtime.instance.Instance.markDead(Instance.java:167)

        at org.apache.flink.runtime.instance.InstanceManager.unregisterTaskManager(InstanceManager.java:214)

        at org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$handleTaskManagerTerminated(JobManager.scala:1160)

        at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:1063)

        at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)

        at org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:44)

        at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)

        at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)

        at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)

        at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)

        at org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)

        at akka.actor.Actor$class.aroundReceive(Actor.scala:465)

        at org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:119)

        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)

        at akka.actor.dungeon.DeathWatch$class.receivedTerminated(DeathWatch.scala:46)

        at akka.actor.ActorCell.receivedTerminated(ActorCell.scala:369)

        at akka.actor.ActorCell.autoReceiveMessage(ActorCell.scala:501)

        at akka.actor.ActorCell.invoke(ActorCell.scala:486)

        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)

        at akka.dispatch.Mailbox.run(Mailbox.scala:221)

        at akka.dispatch.Mailbox.exec(Mailbox.scala:231)

        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)

        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)

        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)

        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

2016-11-25 07:19:58,136 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       
- Could not restart the job shop-monitor (cd3b18a4854c3f720cb581b1c84830c4).

 

//task manager

2016-11-25 07:08:31,312 INFO  org.apache.zookeeper.ZooKeeper                               
- Initiating client connection, connectString=kafka.zk1.service.mogujie.org:2181,kafka.zk2.service.mogujie.org:2181,kafka.zk3.service.mogujie.org:2181
sessionTimeout=60000 watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@77f9e968

2016-11-25 07:08:31,319 WARN  org.apache.flink.shaded.org.apache.curator.ConnectionState   
- Connection attempt unsuccessful after 147624 (greater than max timeout of 60000). Resetting
connection and trying again with a new connection.

2016-11-25 07:08:31,321 INFO  org.apache.zookeeper.ClientCnxn                              
- Opening socket connection to server 10.17.34.11/10.17.34.11:2181

2016-11-25 07:08:31,322 INFO  org.apache.zookeeper.ClientCnxn                               -
Socket connection established to 10.17.34.11/10.17.34.11:2181, initiating session

2016-11-25 07:08:31,325 INFO  org.apache.zookeeper.ClientCnxn                              
- Session establishment complete on server 10.17.34.11/10.17.34.11:2181, sessionid = 0x456b80d2f6ce4c7,
negotiated timeout = 40000

2016-11-25 07:09:45,169 INFO  org.apache.zookeeper.ZooKeeper                               
- Session: 0x456b80d2f6ce4c4 closed

2016-11-25 07:09:45,170 INFO  org.apache.zookeeper.ClientCnxn                              
- EventThread shut down

2016-11-25 07:09:45,170 INFO  org.apache.zookeeper.ZooKeeper                               
- Session: 0x0 closed

2016-11-25 07:09:45,169 INFO  org.apache.zookeeper.ClientCnxn                       
       - EventThread shut down

2016-11-25 07:09:45,170 INFO  org.apache.zookeeper.ZooKeeper                               
- Initiating client connection, connectString=kafka.zk1.service.mogujie.org:2181,kafka.zk2.service.mogujie.org:2181,kafka.zk3.service.mogujie.org:2181
sessionTimeout=60000 watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@3a5d2ba6

2016-11-25 07:09:45,170 INFO  org.apache.zookeeper.ZooKeeper                               
- Initiating client connection, connectString=kafka.zk1.service.mogujie.org:2181,kafka.zk2.service.mogujie.org:2181,kafka.zk3.service.mogujie.org:2181
sessionTimeout=60000 watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@50d1bcf2

2016-11-25 07:09:45,171 INFO  com.mogujie.corgi.common.keeper.KeeperProxy                  
- unable to refresh keeper status, cause: java.util.concurrent.TimeoutException, master: 10.15.2.123:8888

2016-11-25 07:09:45,171 WARN  com.mogujie.corgi.net.handler.DispatchHandler                
- no future for response, route: 1, from: /10.15.2.123:8888, packetId: 17073

2016-11-25 07:09:45,171 WARN  org.apache.flink.shaded.org.apache.curator.ConnectionState   
- Connection attempt unsuccessful after 147297 (greater than max timeout of 60000). Resetting
connection and trying again with a new connection.

2016-11-25 07:09:45,174 WARN  org.apache.flink.shaded.org.apache.curator.ConnectionState   
- Connection attempt unsuccessful after 147300 (greater than max timeout of 60000). Resetting
connection and trying again with a new connection.

2016-11-25 07:09:45,174 ERROR org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl 
- Background operation retry gave up

org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss

        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)

        at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:708)

        at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:826)

        at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:792)

        at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62)

        at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

2016-11-25 07:09:45,175 INFO  org.apache.zookeeper.ClientCnxn                              
- Opening socket connection to server 10.17.36.74/10.17.36.74:2181

2016-11-25 07:09:45,176 INFO  org.apache.zookeeper.ClientCnxn                              
- Opening socket connection to server 10.17.34.11/10.17.34.11:2181

2016-11-25 07:09:45,176 ERROR org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl 
- Background retry gave up

org.apache.flink.shaded.org.apache.curator.CuratorConnectionLossException: KeeperErrorCode
= ConnectionLoss

        at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:809)

        at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:792)

        at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62)

        at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

2016-11-25 07:09:45,177 INFO  org.apache.zookeeper.ClientCnxn                               -
Socket connection established to 10.17.36.74/10.17.36.74:2181, initiating session

2016-11-25 07:09:45,177 INFO  org.apache.zookeeper.ClientCnxn                              
- Socket connection established to 10.17.34.11/10.17.34.11:2181, initiating session

2016-11-25 07:09:45,179 INFO  org.apache.zookeeper.ClientCnxn                              
- Session establishment complete on server 10.17.36.74/10.17.36.74:2181, sessionid = 0x556b80d3a88e2f1,
negotiated timeout = 40000

2016-11-25 07:09:45,179 INFO  org.apache.zookeeper.ClientCnxn                              
- Session establishment complete on server 10.17.34.11/10.17.34.11:2181, sessionid = 0x456b80d2f6ce4cc,
negotiated timeout = 40000

2016-11-25 07:09:45,180 INFO  org.apache.zookeeper.ClientCnxn                              
- EventThread shut down

2016-11-25 07:09:45,180 INFO  org.apache.zookeeper.ZooKeeper                               
- Session: 0x556b80d3a88e2f1 closed

2016-11-25 07:09:45,181 INFO  org.apache.zookeeper.ZooKeeper                               
- Initiating client connection, connectString=kafka.zk1.service.mogujie.org:2181,kafka.zk2.service.mogujie.org:2181,kafka.zk3.service.mogujie.org:2181
sessionTimeout=60000 watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@3a5d2ba6

2016-11-25 07:09:45,181 INFO  org.apache.zookeeper.ZooKeeper                               
- Session: 0x456b80d2f6ce4cc closed

2016-11-25 07:09:45,181 INFO  org.apache.zookeeper.ClientCnxn                              
- EventThread shut down

2016-11-25 07:09:45,181 INFO  org.apache.zookeeper.ZooKeeper                               
- Initiating client connection, connectString=kafka.zk1.service.mogujie.org:2181,kafka.zk2.service.mogujie.org:2181,kafka.zk3.service.mogujie.org:2181
sessionTimeout=60000 watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@50d1bcf2

2016-11-25 07:09:45,182 INFO  org.apache.flink.shaded.org.apache.curator.framework.state.ConnectionStateManager 
- State change: LOST

2016-11-25 07:10:59,160 INFO  org.apache.zookeeper.ZooKeeper                               
- Session: 0x456b80d2f6ce4c7 closed

2016-11-25 07:10:59,163 INFO  org.apache.zookeeper.ClientCnxn                              
- Opening socket connection to server 10.17.34.11/10.17.34.11:2181

2016-11-25 07:10:59,161 INFO  org.apache.zookeeper.ClientCnxn                              
- Opening socket connection to server 10.17.34.11/10.17.34.11:2181

2016-11-25 07:10:59,161 ERROR org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl 
- Background operation retry gave up

org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss

        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)

        at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:708)

        at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:826)

        at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:792)

        at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62)

        at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

2016-11-25 07:10:59,161 ERROR org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl 
- Background operation retry gave up

org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss

        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)

        at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:708)

        at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:826)

        at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:792)

        at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62)

        at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

2016-11-25 07:10:59,160 INFO  org.apache.zookeeper.ClientCnxn                              
- EventThread shut down

2016-11-25 07:10:59,167 ERROR org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl 
- Background retry gave up

org.apache.flink.shaded.org.apache.curator.CuratorConnectionLossException: KeeperErrorCode
= ConnectionLoss

        at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:809)

        at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:792)

        at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62)

        at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

2016-11-25 07:10:59,167 INFO  org.apache.zookeeper.ClientCnxn                              
- Socket connection established to 10.17.34.11/10.17.34.11:2181, initiating session

2016-11-25 07:10:59,167 ERROR org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl 
- Background retry gave up

org.apache.flink.shaded.org.apache.curator.CuratorConnectionLossException: KeeperErrorCode
= ConnectionLoss

        at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:809)

        at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:792)

        at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62)

        at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

2016-11-25 07:10:59,168 WARN  org.apache.flink.shaded.org.apache.curator.ConnectionState   
- Connection attempt unsuccessful after 73997 (greater than max timeout of 60000). Resetting
connection and trying again with a new connection.

2016-11-25 07:10:59,167 INFO  org.apache.zookeeper.ClientCnxn                              
- Client session timed out, have not heard from server in 73985ms for sessionid 0x0, closing
socket connection and attempting reconnect

2016-11-25 07:10:59,163 INFO  org.apache.zookeeper.ZooKeeper                               
- Initiating client connection, connectString=kafka.zk1.service.mogujie.org:2181,kafka.zk2.service.mogujie.org:2181,kafka.zk3.service.mogujie.org:2181
sessionTimeout=60000 watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@77f9e968

2016-11-25 07:10:59,168 WARN  org.apache.flink.shaded.org.apache.curator.ConnectionState   
- Connection attempt unsuccessful after 73994 (greater than max timeout of 60000). Resetting
connection and trying again with a new connection.

2016-11-25 07:10:59,169 INFO  org.apache.zookeeper.ClientCnxn                              
- Session establishment complete on server 10.17.34.11/10.17.34.11:2181, sessionid = 0x456b80d2f6ce4ce,
negotiated timeout = 40000

2016-11-25 07:12:13,370 WARN  org.apache.flink.shaded.org.apache.curator.ConnectionState   
- Connection attempt unsuccessful after 147852 (greater than max timeout of 60000). Resetting
connection and trying again with a new connection.

2016-11-25 07:12:13,370 INFO  org.apache.zookeeper.ClientCnxn                              
- Opening socket connection to server 10.17.34.22/10.17.34.22:2181

2016-11-25 07:12:13,370 WARN  com.mogujie.corgi.net.handler.DispatchHandler                
- no future for response, route: 1, from: /10.11.13.22:9003, packetId: 17076

2016-11-25 07:12:13,373 INFO  org.apache.zookeeper.ClientCnxn                              
- EventThread shut down

2016-11-25 07:12:13,373 INFO  com.mogujie.corgi.net.channel.AbstractChannelHandler         
- user idleTriggered event triggered, channel: [id: 0x60449ebe, /10.17.123.56:14660 =>
corgi.keeper.service.mogujie.org/10.15.2.123:8888]

2016-11-25 07:12:13,373 INFO  org.apache.zookeeper.ZooKeeper                               
- Session: 0x0 closed

2016-11-25 07:12:13,373 INFO  org.apache.zookeeper.ZooKeeper                               
- Initiating client connection, connectString=kafka.zk1.service.mogujie.org:2181,kafka.zk2.service.mogujie.org:2181,kafka.zk3.service.mogujie.org:2181
sessionTimeout=60000 watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@50d1bcf2

2016-11-25 07:12:13,375 WARN  org.apache.flink.shaded.org.apache.curator.ConnectionState   
- Connection attempt unsuccessful after 74205 (greater than max timeout of 60000). Resetting
connection and trying again with a new connection.

2016-11-25 07:12:13,377 INFO  org.apache.zookeeper.ClientCnxn                              
- Opening socket connection to server 10.17.34.22/10.17.34.22:2181

2016-11-25 07:12:13,378 INFO  org.apache.zookeeper.ClientCnxn                              
- Socket connection established to 10.17.34.22/10.17.34.22:2181, initiating session

2016-11-25 07:13:27,110 INFO  org.apache.zookeeper.ZooKeeper                               
- Session: 0x456b80d2f6ce4ce closed

2016-11-25 07:13:27,110 INFO  org.apache.zookeeper.ClientCnxn                              
- EventThread shut down

2016-11-25 07:13:27,111 INFO  org.apache.zookeeper.ZooKeeper                               
- Initiating client connection, connectString=kafka.zk1.service.mogujie.org:2181,kafka.zk2.service.mogujie.org:2181,kafka.zk3.service.mogujie.org:2181
sessionTimeout=60000 watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@3a5d2ba6

2016-11-25 07:13:27,111 INFO  org.apache.zookeeper.ZooKeeper                               
- Session: 0x0 closed

2016-11-25 07:13:27,111 INFO  org.apache.zookeeper.ZooKeeper                               
- Initiating client connection, connectString=kafka.zk1.service.mogujie.org:2181,kafka.zk2.service.mogujie.org:2181,kafka.zk3.service.mogujie.org:2181
sessionTimeout=60000 watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@77f9e968

2016-11-25 07:13:27,111 INFO  org.apache.zookeeper.ClientCnxn                 
             - EventThread shut down

2016-11-25 07:13:27,112 WARN  org.apache.flink.shaded.org.apache.curator.ConnectionState   
- Connection attempt unsuccessful after 147944 (greater than max timeout of 60000). Resetting
connection and trying again with a new connection.

2016-11-25 07:13:27,112 WARN  org.apache.flink.shaded.org.apache.curator.ConnectionState   
- Connection attempt unsuccessful after 73742 (greater than max timeout of 60000). Resetting
connection and trying again with a new connection.

2016-11-25 07:13:27,114 INFO  org.apache.zookeeper.ClientCnxn                              
- Opening socket connection to server 10.17.34.22/10.17.34.22:2181

2016-11-25 07:13:27,114 INFO  org.apache.zookeeper.ClientCnxn                              
- Opening socket connection to server 10.17.34.11/10.17.34.11:2181

2016-11-25 07:13:27,115 INFO  org.apache.zookeeper.ClientCnxn                              
- Socket connection established to 10.17.34.22/10.17.34.22:2181, initiating session

2016-11-25 07:13:27,117 INFO  org.apache.zookeeper.ClientCnxn                              
- Socket connection established to 10.17.34.11/10.17.34.11:2181, initiating session

2016-11-25 07:13:27,118 INFO  org.apache.zookeeper.ClientCnxn                              
- Session establishment complete on server 10.17.34.22/10.17.34.22:2181, sessionid = 0x356b80d2eebe879,
negotiated timeout = 40000

2016-11-25 07:13:27,118 INFO  org.apache.zookeeper.ClientCnxn                              
- Session establishment complete on server 10.17.34.11/10.17.34.11:2181, sessionid = 0x456b80d2f6ce4da,
negotiated timeout = 40000

2016-11-25 07:14:33,247 INFO  org.apache.zookeeper.ZooKeeper                               
- Session: 0x356b80d2eebe879 closed

2016-11-25 07:14:33,248 INFO  org.apache.zookeeper.ZooKeeper                               
- Session: 0x0 closed

2016-11-25 07:14:33,248 INFO  org.apache.zookeeper.ClientCnxn                              
- EventThread shut down

2016-11-25 07:14:33,247 INFO  org.apache.zookeeper.ClientCnxn                               -
EventThread shut down

2016-11-25 07:14:33,248 INFO  org.apache.zookeeper.ZooKeeper                               
- Initiating client connection, connectString=kafka.zk1.service.mogujie.org:2181,kafka.zk2.service.mogujie.org:2181,kafka.zk3.service.mogujie.org:2181
sessionTimeout=60000 watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@50d1bcf2

2016-11-25 07:14:33,248 INFO  org.apache.zookeeper.ZooKeeper                               
- Initiating client connection, connectString=kafka.zk1.service.mogujie.org:2181,kafka.zk2.service.mogujie.org:2181,kafka.zk3.service.mogujie.org:2181
sessionTimeout=60000 watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@3a5d2ba6

2016-11-25 07:14:33,249 WARN  org.apache.flink.shaded.org.apache.curator.ConnectionState   
- Connection attempt unsuccessful after 66137 (greater than max timeout of 60000). Resetting
connection and trying again with a new connection.

2016-11-25 07:14:33,249 WARN  org.apache.flink.shaded.org.apache.curator.ConnectionState   
- Connection attempt unsuccessful after 139874 (greater than max timeout of 60000). Resetting
connection and trying again with a new connection.

2016-11-25 07:14:33,253 INFO  org.apache.zookeeper.ClientCnxn                               -
Opening socket connection to server 10.17.34.22/10.17.34.22:2181

2016-11-25 07:14:33,253 INFO  org.apache.zookeeper.ClientCnxn                              
- Socket connection established to 10.17.34.22/10.17.34.22:2181, initiating session

2016-11-25 07:14:33,255 INFO  org.apache.zookeeper.ClientCnxn                              
- Session establishment complete on server 10.17.34.22/10.17.34.22:2181, sessionid = 0x356b80d2eebe87d,
negotiated timeout = 40000

2016-11-25 07:14:33,256 INFO  org.apache.zookeeper.ClientCnxn                              
- EventThread shut down

2016-11-25 07:14:33,256 INFO  org.apache.zookeeper.ZooKeeper                               
- Session: 0x356b80d2eebe87d closed

2016-11-25 07:14:33,258 INFO  org.apache.zookeeper.ZooKeeper                               
- Initiating client connection, connectString=kafka.zk1.service.mogujie.org:2181,kafka.zk2.service.mogujie.org:2181,kafka.zk3.service.mogujie.org:2181
sessionTimeout=60000 watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@50d1bcf2

2016-11-25 07:15:38,952 WARN  org.apache.flink.shaded.org.apache.curator.ConnectionState   
- Connection attempt unsuccessful after 65703 (greater than max timeout of 60000). Resetting
connection and trying again with a new connection.

2016-11-25 07:15:38,953 INFO  org.apache.zookeeper.ClientCnxn                              
- Opening socket connection to server 10.17.36.74/10.17.36.74:2181

2016-11-25 07:15:38,953 INFO  org.apache.zookeeper.ClientCnxn                               -
Opening socket connection to server 10.17.34.22/10.17.34.22:2181

2016-11-25 07:15:38,955 INFO  org.apache.zookeeper.ZooKeeper                               
- Session: 0x456b80d2f6ce4da closed

2016-11-25 07:15:38,955 INFO  org.apache.zookeeper.ClientCnxn                              
- EventThread shut down

2016-11-25 07:15:38,955 INFO  org.apache.zookeeper.ZooKeeper                               
- Initiating client connection, connectString=kafka.zk1.service.mogujie.org:2181,kafka.zk2.service.mogujie.org:2181,kafka.zk3.service.mogujie.org:2181
sessionTimeout=60000 watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@77f9e968

2016-11-25 07:16:45,848 WARN  org.apache.flink.shaded.org.apache.curator.ConnectionState   
- Connection attempt unsuccessful after 131846 (greater than max timeout of 60000). Resetting
connection and trying again with a new connection.

2016-11-25 07:16:45,850 INFO  org.apache.zookeeper.ClientCnxn                              
- Opening socket connection to server 10.17.36.74/10.17.36.74:2181

2016-11-25 07:16:45,850 INFO  org.apache.zookeeper.ClientCnxn                              
- EventThread shut down

2016-11-25 07:16:45,850 INFO  org.apache.zookeeper.ZooKeeper                               
- Session: 0x0 closed

2016-11-25 07:16:45,850 INFO  org.apache.zookeeper.ClientCnxn                              
- EventThread shut down

2016-11-25 07:16:45,850 INFO  org.apache.zookeeper.ZooKeeper                               
- Session: 0x0 closed

2016-11-25 07:16:45,850 WARN  com.mogujie.corgi.net.handler.DispatchHandler                
- no future for response, route: 1, from: /10.17.36.202:9003, packetId: 17095

2016-11-25 07:16:45,852 INFO  org.apache.zookeeper.ZooKeeper                               
- Initiating client connection, connectString=kafka.zk1.service.mogujie.org:2181,kafka.zk2.service.mogujie.org:2181,kafka.zk3.service.mogujie.org:2181
sessionTimeout=60000 watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@50d1bcf2

2016-11-25 07:16:45,851 INFO  org.apache.zookeeper.ZooKeeper                               
- Initiating client connection, connectString=kafka.zk1.service.mogujie.org:2181,kafka.zk2.service.mogujie.org:2181,kafka.zk3.service.mogujie.org:2181
sessionTimeout=60000 watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@3a5d2ba6

2016-11-25 07:16:45,853 WARN  org.apache.flink.shaded.org.apache.curator.ConnectionState   
- Connection attempt unsuccessful after 66900 (greater than max timeout of 60000). Resetting
connection and trying again with a new connection.

2016-11-25 07:16:45,853 WARN  org.apache.flink.shaded.org.apache.curator.ConnectionState   
- Connection attempt unsuccessful after 132604 (greater than max timeout of 60000). Resetting
connection and trying again with a new connection.

2016-11-25 07:16:45,855 INFO  org.apache.zookeeper.ClientCnxn                              
- Opening socket connection to server 10.17.36.74/10.17.36.74:2181

2016-11-25 07:16:45,856 INFO  org.apache.zookeeper.ClientCnxn                               -
Socket connection established to 10.17.36.74/10.17.36.74:2181, initiating session

2016-11-25 07:19:05,172 INFO  org.apache.zookeeper.ZooKeeper                               
- Session: 0x0 closed

2016-11-25 07:21:32,016 INFO  org.apache.zookeeper.ClientCnxn                              
- Opening socket connection to server 10.17.36.74/10.17.36.74:2181

2016-11-25 07:21:32,016 INFO  org.apache.zookeeper.ClientCnxn                              
- EventThread shut down

2016-11-25 07:21:32,016 INFO  org.apache.zookeeper.ZooKeeper                               
- Session: 0x0 closed

2016-11-25 07:19:05,172 INFO  org.apache.zookeeper.ClientCnxn                              
- EventThread shut down

2016-11-25 07:25:16,838 INFO  com.mogujie.corgi.net.channel.AbstractChannelHandler         
- user idleTriggered event triggered, channel: [id: 0x60449ebe, /10.17.123.56:14660 =>
corgi.keeper.service.mogujie.org/10.15.2.123:8888]

2016-11-25 07:24:02,740 INFO  org.apache.zookeeper.ZooKeeper                                -
Initiating client connection, connectString=kafka.zk1.service.mogujie.org:2181,kafka.zk2.service.mogujie.org:2181,kafka.zk3.service.mogujie.org:2181
sessionTimeout=60000 watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@77f9e968

2016-11-25 07:27:43,909 INFO  org.apache.zookeeper.ZooKeeper                               
- Initiating client connection, connectString=kafka.zk1.service.mogujie.org:2181,kafka.zk2.service.mogujie.org:2181,kafka.zk3.service.mogujie.org:2181
sessionTimeout=60000 watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@3a5d2ba6

2016-11-25 07:27:43,909 INFO  com.mogujie.corgi.net.channel.AbstractChannelHandler         
- user idleTriggered event triggered, channel: [id: 0xcf39e7ef, /10.17.123.56:31625 =>
/10.11.13.22:9003]

2016-11-25 07:27:43,910 INFO  com.mogujie.corgi.net.channel.AbstractChannelHandler         
- user idleTriggered event triggered, channel: [id: 0x12f19cbd, /10.17.123.56:18394 =>
/10.11.13.14:9003]

 

I suppose there are some bugs cause this error.


Mime
View raw message