flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joshua Fan <joshuafat...@gmail.com>
Subject Re: Fencing token exceptions from Job Manager High Availability mode
Date Thu, 10 Oct 2019 09:31:05 GMT
*Hi Till*

*Recently, this issue happened frequently, but some jobs recovered from the
exception, some not.*

*for the job who can not recover from the exception, the log pasted below:*

2019-10-09 13:54:21.746 [main-EventThread] INFO
o.a.f.s.c.o.a.curator.framework.state.ConnectionStateManager  - State
change: SUSPENDED
2019-10-09 13:54:21.746 [Curator-ConnectionStateManager-0] WARN
o.a.f.runtime.leaderelection.ZooKeeperLeaderElectionService  -
Connection to ZooKeeper suspended. The contender
akka.tcp://flink@10.162.89.153:16736/user/dispatcher no longer
participates in the leader election.
2019-10-09 13:54:21.746 [flink-akka.actor.default-dispatcher-46] INFO
org.apache.flink.yarn.YarnResourceManager  - ResourceManager
akka.tcp://flink@10.162.89.153:16736/user/resourcemanager was revoked
leadership. Clearing fencing token.
2019-10-09 13:54:21.746 [Curator-ConnectionStateManager-0] WARN
o.a.f.runtime.leaderelection.ZooKeeperLeaderElectionService  -
Connection to ZooKeeper suspended. The contender
http://10.162.89.153:12503 no longer participates in the leader
election.
2019-10-09 13:54:21.747 [Curator-ConnectionStateManager-0] INFO
o.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint  -
http://10.162.89.153:12503 lost leadership
2019-10-09 13:54:21.747 [Curator-ConnectionStateManager-0] WARN
o.a.f.runtime.leaderelection.ZooKeeperLeaderElectionService  -
Connection to ZooKeeper suspended. The contender
akka.tcp://flink@10.162.89.153:16736/user/resourcemanager no longer
participates in the leader election.
2019-10-09 13:54:21.747 [Curator-ConnectionStateManager-0] WARN
o.a.f.r.leaderretrieval.ZooKeeperLeaderRetrievalService  - Connection
to ZooKeeper suspended. Can no longer retrieve the leader from
ZooKeeper.
2019-10-09 13:54:21.747 [Curator-ConnectionStateManager-0] WARN
o.a.f.r.leaderretrieval.ZooKeeperLeaderRetrievalService  - Connection
to ZooKeeper suspended. Can no longer retrieve the leader from
ZooKeeper.
2019-10-09 13:54:21.747 [flink-akka.actor.default-dispatcher-43] INFO
org.apache.flink.runtime.dispatcher.MiniDispatcher  - Dispatcher
akka.tcp://flink@10.162.89.153:16736/user/dispatcher was revoked
leadership.
2019-10-09 13:54:21.747 [flink-akka.actor.default-dispatcher-46] INFO
o.a.flink.runtime.resourcemanager.slotmanager.SlotManager  -
Suspending the SlotManager.
2019-10-09 13:54:22.072 [main-SendThread(10.160.101.26:2181)] WARN
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL
configuration failed: javax.security.auth.login.LoginException: No
JAAS configuration section named 'Client' was found in specified JAAS
configuration file: '/tmp/jaas-1334102078344377682.conf'. Will
continue connection to Zookeeper server without SASL authentication,
if Zookeeper server allows it.
2019-10-09 13:54:22.072 [main-SendThread(10.160.101.26:2181)] INFO
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Opening
socket connection to server 10.160.101.26/10.160.101.26:2181
2019-10-09 13:54:22.072 [main-EventThread] ERROR
o.a.flink.shaded.curator.org.apache.curator.ConnectionState  -
Authentication failed
2019-10-09 13:54:22.072 [main-SendThread(10.160.101.26:2181)] WARN
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Session
0x2597758ae5091ff for server null, unexpected error, closing socket
connection and attempting reconnect
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
	at org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
2019-10-09 13:54:22.973 [main-SendThread(10.160.101.27:2181)] WARN
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL
configuration failed: javax.security.auth.login.LoginException: No
JAAS configuration section named 'Client' was found in specified JAAS
configuration file: '/tmp/jaas-1334102078344377682.conf'. Will
continue connection to Zookeeper server without SASL authentication,
if Zookeeper server allows it.
2019-10-09 13:54:22.973 [main-SendThread(10.160.101.27:2181)] INFO
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Opening
socket connection to server 10.160.101.27/10.160.101.27:2181
2019-10-09 13:54:22.973 [main-EventThread] ERROR
o.a.flink.shaded.curator.org.apache.curator.ConnectionState  -
Authentication failed
2019-10-09 13:54:22.974 [main-SendThread(10.160.101.27:2181)] WARN
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Session
0x2597758ae5091ff for server null, unexpected error, closing socket
connection and attempting reconnect
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
	at org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
2019-10-09 13:54:23.405 [main-SendThread(10.160.101.30:2181)] WARN
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL
configuration failed: javax.security.auth.login.LoginException: No
JAAS configuration section named 'Client' was found in specified JAAS
configuration file: '/tmp/jaas-1334102078344377682.conf'. Will
continue connection to Zookeeper server without SASL authentication,
if Zookeeper server allows it.
2019-10-09 13:54:23.405 [main-SendThread(10.160.101.30:2181)] INFO
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Opening
socket connection to server 10.160.101.30/10.160.101.30:2181
2019-10-09 13:54:23.405 [main-EventThread] ERROR
o.a.flink.shaded.curator.org.apache.curator.ConnectionState  -
Authentication failed
2019-10-09 13:54:23.406 [main-SendThread(10.160.101.30:2181)] WARN
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Session
0x2597758ae5091ff for server null, unexpected error, closing socket
connection and attempting reconnect
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
	at org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
2019-10-09 13:54:24.379 [main-SendThread(10.160.101.28:2181)] WARN
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL
configuration failed: javax.security.auth.login.LoginException: No
JAAS configuration section named 'Client' was found in specified JAAS
configuration file: '/tmp/jaas-1334102078344377682.conf'. Will
continue connection to Zookeeper server without SASL authentication,
if Zookeeper server allows it.
2019-10-09 13:54:24.379 [main-SendThread(10.160.101.28:2181)] INFO
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Opening
socket connection to server 10.160.101.28/10.160.101.28:2181
2019-10-09 13:54:24.379 [main-EventThread] ERROR
o.a.flink.shaded.curator.org.apache.curator.ConnectionState  -
Authentication failed
2019-10-09 13:54:24.380 [main-SendThread(10.160.101.28:2181)] WARN
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Session
0x2597758ae5091ff for server null, unexpected error, closing socket
connection and attempting reconnect
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
	at org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
2019-10-09 13:54:26.044 [main-SendThread(10.160.101.29:2181)] WARN
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL
configuration failed: javax.security.auth.login.LoginException: No
JAAS configuration section named 'Client' was found in specified JAAS
configuration file: '/tmp/jaas-1334102078344377682.conf'. Will
continue connection to Zookeeper server without SASL authentication,
if Zookeeper server allows it.
2019-10-09 13:54:26.045 [main-SendThread(10.160.101.29:2181)] INFO
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Opening
socket connection to server 10.160.101.29/10.160.101.29:2181
2019-10-09 13:54:26.045 [main-EventThread] ERROR
o.a.flink.shaded.curator.org.apache.curator.ConnectionState  -
Authentication failed
2019-10-09 13:54:29.050 [main-SendThread(10.160.101.29:2181)] INFO
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Socket
connection established to 10.160.101.29/10.160.101.29:2181, initiating
session
2019-10-09 13:54:29.095 [main-SendThread(10.160.101.29:2181)] INFO
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Session
establishment complete on server 10.160.101.29/10.160.101.29:2181,
sessionid = 0x2597758ae5091ff, negotiated timeout = 40000
2019-10-09 13:54:29.095 [main-EventThread] INFO
o.a.f.s.c.o.a.curator.framework.state.ConnectionStateManager  - State
change: RECONNECTED
2019-10-09 13:54:29.096 [Curator-ConnectionStateManager-0] INFO
o.a.f.runtime.leaderelection.ZooKeeperLeaderElectionService  -
Connection to ZooKeeper was reconnected. Leader election can be
restarted.
2019-10-09 13:54:29.097 [Curator-ConnectionStateManager-0] INFO
o.a.f.runtime.leaderelection.ZooKeeperLeaderElectionService  -
Connection to ZooKeeper was reconnected. Leader election can be
restarted.
2019-10-09 13:54:29.097 [Curator-ConnectionStateManager-0] INFO
o.a.f.runtime.leaderelection.ZooKeeperLeaderElectionService  -
Connection to ZooKeeper was reconnected. Leader election can be
restarted.
2019-10-09 13:54:29.097 [Curator-ConnectionStateManager-0] INFO
o.a.f.r.leaderretrieval.ZooKeeperLeaderRetrievalService  - Connection
to ZooKeeper was reconnected. Leader retrieval can be restarted.
2019-10-09 13:54:29.097 [Curator-ConnectionStateManager-0] INFO
o.a.f.r.leaderretrieval.ZooKeeperLeaderRetrievalService  - Connection
to ZooKeeper was reconnected. Leader retrieval can be restarted.
2019-10-09 13:54:29.105 [flink-akka.actor.default-dispatcher-46] INFO
org.apache.flink.yarn.YarnResourceManager  - ResourceManager
akka.tcp://flink@10.162.89.153:16736/user/resourcemanager was granted
leadership with fencing token a22e4c9c80fd0c7249102f69c0b64e01
2019-10-09 13:54:29.105 [flink-akka.actor.default-dispatcher-46] INFO
o.a.flink.runtime.resourcemanager.slotmanager.SlotManager  - Starting
the SlotManager.
2019-10-09 13:54:29.107 [main-EventThread] INFO
o.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint  -
http://10.162.89.153:12503 was granted leadership with
leaderSessionID=53b02e00-95c3-4d34-a272-b84795b991a7
2019-10-09 13:54:29.116 [flink-akka.actor.default-dispatcher-26] INFO
org.apache.flink.runtime.dispatcher.MiniDispatcher  - Dispatcher
akka.tcp://flink@10.162.89.153:16736/user/dispatcher was granted
leadership with fencing token d645a3fe-8fee-453b-abd5-bd4cd82ebba1
2019-10-09 13:54:34.809 [flink-rest-server-netty-worker-thread-73]
ERROR org.apache.flink.runtime.rest.handler.job.JobDetailsHandler  -
Could not retrieve the redirect address.
java.util.concurrent.CompletionException:
org.apache.flink.runtime.rpc.exceptions.FencingTokenException: Fencing
token not set: Ignoring message
LocalFencedMessage(9f126af7b71d8b61dcb3687f3dd84166,
LocalRpcInvocation(requestRestAddress(Time))) sent to
akka.tcp://flink@10.162.89.153:16736/user/dispatcher because the
fencing token is null.
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
	at org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:772)
	at akka.dispatch.OnComplete.internal(Future.scala:258)
	at akka.dispatch.OnComplete.internal(Future.scala:256)
	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)
	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
	at org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:83)
	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
	at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:534)
	at akka.actor.ActorRef.tell(ActorRef.scala:130)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.sendErrorIfSender(AkkaRpcActor.java:371)
	at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:57)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:142)
	at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.onReceive(FencedAkkaRpcActor.java:40)
	at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
	at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
	at akka.actor.ActorCell.invoke(ActorCell.scala:495)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
	at akka.dispatch.Mailbox.run(Mailbox.scala:224)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.apache.flink.runtime.rpc.exceptions.FencingTokenException:
Fencing token not set: Ignoring message
LocalFencedMessage(9f126af7b71d8b61dcb3687f3dd84166,
LocalRpcInvocation(requestRestAddress(Time))) sent to
akka.tcp://flink@10.162.89.153:16736/user/dispatcher because the
fencing token is null.
	at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:59)
	... 14 common frames omitted
2019-10-09 13:54:56.007 [flink-rest-server-netty-worker-thread-74]
ERROR org.apache.flink.runtime.rest.handler.job.JobDetailsHandler  -
Could not retrieve the redirect address.
java.util.concurrent.CompletionException:
org.apache.flink.runtime.rpc.exceptions.FencingTokenException: Fencing
token not set: Ignoring message
LocalFencedMessage(9f126af7b71d8b61dcb3687f3dd84166,
LocalRpcInvocation(requestRestAddress(Time))) sent to
akka.tcp://flink@10.162.89.153:16736/user/dispatcher because the
fencing token is null.
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
	at org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:772)
	at akka.dispatch.OnComplete.internal(Future.scala:258)
	at akka.dispatch.OnComplete.internal(Future.scala:256)
	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)
	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
	at org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:83)
	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
	at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:534)
	at akka.actor.ActorRef.tell(ActorRef.scala:130)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.sendErrorIfSender(AkkaRpcActor.java:371)
	at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:57)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:142)
	at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.onReceive(FencedAkkaRpcActor.java:40)
	at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
	at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
	at akka.actor.ActorCell.invoke(ActorCell.scala:495)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
	at akka.dispatch.Mailbox.run(Mailbox.scala:224)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.apache.flink.runtime.rpc.exceptions.FencingTokenException:
Fencing token not set: Ignoring message
LocalFencedMessage(9f126af7b71d8b61dcb3687f3dd84166,
LocalRpcInvocation(requestRestAddress(Time))) sent to
akka.tcp://flink@10.162.89.153:16736/user/dispatcher because the
fencing token is null.
	at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:59)
	... 14 common frames omitted


*In the above log, the ha zk got reconnected after a suspend, but the
job can not recover, and there existed a MiniDispatcherRestEndpoint.*



*for the job who can recover from the same condition, the log pasted as below:*


2019-10-09 13:54:21.803 [main-EventThread] INFO
o.a.f.s.c.o.a.curator.framework.state.ConnectionStateManager  - State
change: SUSPENDED
2019-10-09 13:54:21.804 [Curator-ConnectionStateManager-0] WARN
o.a.f.runtime.leaderelection.ZooKeeperLeaderElectionService  -
Connection to ZooKeeper suspended. The contender
http://10.160.140.216:52303 no longer participates in the leader
election.
2019-10-09 13:54:21.804 [Curator-ConnectionStateManager-0] INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint  -
http://10.160.140.216:52303 lost leadership
2019-10-09 13:54:21.804 [Curator-PathChildrenCache-0] WARN
o.a.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore  -
ZooKeeper connection SUSPENDING. Changes to the submitted job graphs
are not monitored (temporarily).
2019-10-09 13:54:21.805 [Curator-ConnectionStateManager-0] WARN
o.a.f.r.leaderretrieval.ZooKeeperLeaderRetrievalService  - Connection
to ZooKeeper suspended. Can no longer retrieve the leader from
ZooKeeper.
2019-10-09 13:54:21.805 [flink-akka.actor.default-dispatcher-88] INFO
org.apache.flink.yarn.YarnResourceManager  - ResourceManager
akka.tcp://flink@10.160.140.216:26073/user/resourcemanager was revoked
leadership. Clearing fencing token.
2019-10-09 13:54:21.805 [Curator-ConnectionStateManager-0] INFO
org.apache.flink.runtime.jobmaster.JobManagerRunner  - JobManager for
job hdp-shoujiserver_CommercialDotingAccess
(2657b25ff073a91c888ff78a28bd2331) was revoked leadership at
akka.tcp://flink@10.160.140.216:26073/user/jobmanager_2.
2019-10-09 13:54:21.805 [flink-akka.actor.default-dispatcher-88] INFO
o.a.f.r.leaderretrieval.ZooKeeperLeaderRetrievalService  - Stopping
ZooKeeperLeaderRetrievalService
/leader/2657b25ff073a91c888ff78a28bd2331/job_manager_lock.
2019-10-09 13:54:21.806 [Curator-ConnectionStateManager-0] WARN
o.a.f.runtime.leaderelection.ZooKeeperLeaderElectionService  -
Connection to ZooKeeper suspended. The contender
akka.tcp://flink@10.160.140.216:26073/user/jobmanager_2 no longer
participates in the leader election.
2019-10-09 13:54:21.806 [flink-akka.actor.default-dispatcher-20] INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher  - Dispatcher
akka.tcp://flink@10.160.140.216:26073/user/dispatcher was revoked
leadership.
2019-10-09 13:54:21.806 [flink-akka.actor.default-dispatcher-80] INFO
o.a.f.r.leaderretrieval.ZooKeeperLeaderRetrievalService  - Stopping
ZooKeeperLeaderRetrievalService /leader/resource_manager_lock.
2019-10-09 13:54:21.806 [flink-akka.actor.default-dispatcher-20] INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher  - Stopping
all currently running jobs of dispatcher
akka.tcp://flink@10.160.140.216:26073/user/dispatcher.
2019-10-09 13:54:21.806 [Curator-ConnectionStateManager-0] WARN
o.a.f.runtime.leaderelection.ZooKeeperLeaderElectionService  -
Connection to ZooKeeper suspended. The contender
akka.tcp://flink@10.160.140.216:26073/user/dispatcher no longer
participates in the leader election.
2019-10-09 13:54:21.808 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - Job
hdp-shoujiserver_CommercialDotingAccess
(2657b25ff073a91c888ff78a28bd2331) switched from state RUNNING to
SUSPENDING.
org.apache.flink.util.FlinkException: JobManager is no longer the leader.
	at org.apache.flink.runtime.jobmaster.JobManagerRunner.revokeLeadership(JobManagerRunner.java:371)
	at org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService.notLeader(ZooKeeperLeaderElectionService.java:247)
	at org.apache.flink.shaded.curator.org.apache.curator.framework.recipes.leader.LeaderLatch$8.apply(LeaderLatch.java:640)
	at org.apache.flink.shaded.curator.org.apache.curator.framework.recipes.leader.LeaderLatch$8.apply(LeaderLatch.java:636)
	at org.apache.flink.shaded.curator.org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93)
	at org.apache.flink.shaded.curator.org.apache.curator.shaded.com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
	at org.apache.flink.shaded.curator.org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:85)
	at org.apache.flink.shaded.curator.org.apache.curator.framework.recipes.leader.LeaderLatch.setLeadership(LeaderLatch.java:635)
	at org.apache.flink.shaded.curator.org.apache.curator.framework.recipes.leader.LeaderLatch.handleStateChange(LeaderLatch.java:623)
	at org.apache.flink.shaded.curator.org.apache.curator.framework.recipes.leader.LeaderLatch.access$000(LeaderLatch.java:64)
	at org.apache.flink.shaded.curator.org.apache.curator.framework.recipes.leader.LeaderLatch$1.stateChanged(LeaderLatch.java:82)
	at org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager$2.apply(ConnectionStateManager.java:259)
	at org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager$2.apply(ConnectionStateManager.java:255)
	at org.apache.flink.shaded.curator.org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93)
	at org.apache.flink.shaded.curator.org.apache.curator.shaded.com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
	at org.apache.flink.shaded.curator.org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:85)
	at org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:253)
	at org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:111)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
2019-10-09 13:54:21.806 [flink-akka.actor.default-dispatcher-88] INFO
o.a.flink.runtime.resourcemanager.slotmanager.SlotManager  -
Suspending the SlotManager.
2019-10-09 13:54:21.808 [Curator-ConnectionStateManager-0] WARN
o.a.f.r.leaderretrieval.ZooKeeperLeaderRetrievalService  - Connection
to ZooKeeper suspended. Can no longer retrieve the leader from
ZooKeeper.
2019-10-09 13:54:21.809 [Curator-ConnectionStateManager-0] WARN
o.a.f.runtime.leaderelection.ZooKeeperLeaderElectionService  -
Connection to ZooKeeper suspended. The contender
akka.tcp://flink@10.160.140.216:26073/user/resourcemanager no longer
participates in the leader election.
2019-10-09 13:54:21.819 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - Source:
WeiShiDataAnalysisSpout (1/8) (4a825def14a88297d19c9e5738dbf668)
switched from RUNNING to CANCELING.
2019-10-09 13:54:21.820 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - Source:
WeiShiDataAnalysisSpout (2/8) (6517a467aaf7be56bfb3821b29852e20)
switched from RUNNING to CANCELING.
2019-10-09 13:54:21.820 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - Source:
WeiShiDataAnalysisSpout (3/8) (7a7ee5b1f3613b97d858564dbfaeb066)
switched from RUNNING to CANCELING.
2019-10-09 13:54:21.820 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - Source:
WeiShiDataAnalysisSpout (4/8) (2d878a7c86e9d6e0bb0fe737a42037e5)
switched from RUNNING to CANCELING.
2019-10-09 13:54:21.821 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - Source:
WeiShiDataAnalysisSpout (5/8) (82b545cdf0918a7d14b8c595884a5caf)
switched from RUNNING to CANCELING.
2019-10-09 13:54:21.821 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - Source:
WeiShiDataAnalysisSpout (6/8) (fb178b7aed87e2fc7c8be11838575cb6)
switched from RUNNING to CANCELING.
2019-10-09 13:54:21.823 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - Source:
WeiShiDataAnalysisSpout (7/8) (7d2956af9eca83b125a5f2e3d65c7454)
switched from RUNNING to CANCELING.
2019-10-09 13:54:21.823 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - Source:
WeiShiDataAnalysisSpout (8/8) (8673406c095fbb33ff63b8116c2dc6c4)
switched from RUNNING to CANCELING.
2019-10-09 13:54:21.823 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(1/60) (fde9e604f69f27dd1e1afc2d6f96d6a6) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.823 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(2/60) (44429fa0423a6c4e2cfa73b7caf98e7f) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.824 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(3/60) (ccb886b9e7f7f3e9391a66a77a95d936) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.824 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(4/60) (9dfbcca1af51c7565d082ae1d4fbca99) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.824 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(5/60) (984560fc276a70ffb3566ec5236a57d6) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.824 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(6/60) (42c8e83ec209f1947dfc53a3451ca002) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.824 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(7/60) (2fff02384094c4889f8d583db6c7c495) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.825 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(8/60) (bd81e085522ef4915a9413517dcb51ec) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.825 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(9/60) (11762d1b367d402bca99be0624d637c6) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.825 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(10/60) (bcba40c174b64e16b1a48e802e0251c8) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.825 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(11/60) (5655a9923af5f1d40436e2531ca87ff9) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.825 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(12/60) (b9e3017ec604802c151939077f6f4f46) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.825 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(13/60) (4fee14cb460433aa6178fe98315631ff) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.826 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(14/60) (ec80588d10b6a11cc52625f95ebf4858) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.826 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(15/60) (5d55b130964c3c164df9938bb9b9b8e9) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.826 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(16/60) (b67388f779efeb6ffcae57aaaf0e992f) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.826 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(17/60) (c91c36736380adc280b7b9235bd26609) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.826 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(18/60) (f658b30a9c2b4dee63de0fb4797d0c3b) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.827 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(19/60) (bb92a17d4bd6279729ac6e9274ca80ee) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.827 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(20/60) (ecf8a4563990fba29df91ee769e5ea2e) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.827 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(21/60) (8ea7047b457ad4bded12ea4dcea1576b) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.827 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(22/60) (a2cea7a46ecffbd5275d355a29bc09f9) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.827 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(23/60) (105456dc1d11fc9d8164ea0d54794d90) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.828 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(24/60) (faa255028f7bba0ef8b832616a09e0cc) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.828 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(25/60) (80a6454bca3392e1050b02c2b689edae) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.828 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(26/60) (de3561bf14ed9617a4727491ad959219) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.828 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(27/60) (585101b822efc77260c0f0a52eb5f856) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.829 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(28/60) (3205706648aa73f9e34d29c22428ff90) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.829 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(29/60) (24dd134cb75696784bade727d6c57459) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.829 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(30/60) (7d21e0c3934c4bfdc5349f0627fdd923) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.829 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(31/60) (07672677de03585cc630e1014143bec8) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.829 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(32/60) (72159fe698efbaacfb4690b22e4dad03) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.830 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(33/60) (a2f3dce63aac51c82060f02646019af5) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.830 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(34/60) (c5e4f81b8c2590d0f00c7a5c3c05f7a1) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.830 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(35/60) (87579d8547b643a5e1418dca333b426f) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.830 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(36/60) (021f60ef60686370a2e9e6d92fa033ad) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.837 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(37/60) (65b2647fabcbfb1b23aaea63d4627509) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.837 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(38/60) (3f054f2dc5161bd67bbec03d711dea18) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.837 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(39/60) (9b7f93456479e1943de63da8ff8c88d3) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.837 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(40/60) (f2c204e511ed878b8bbd9fded928c215) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.838 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(41/60) (e139c0820ecd8cb8c8fe168b9a847a1e) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.838 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(42/60) (42b961618e0fe859f0761e50e0569113) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.838 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(43/60) (29928821d6f94e715cc2ab444880fb7b) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.838 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(44/60) (83e9b131ba8684474e13236b67916ac3) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.838 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(45/60) (fe3e5b53d7fbc4e795822944317e85bb) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.838 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(46/60) (d7617e20894c359f95cd363faaa3b117) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.839 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(47/60) (e7545083f44b6a8338750542943d67f0) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.839 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(48/60) (0fc95f3ca9dafaeb8d90cd262de82440) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.839 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(49/60) (aa41f35f708625f08bde4b3513e3d769) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.839 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(50/60) (4d01c8d2543d86f2cdd89db1b81df2af) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.839 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(51/60) (6d5314d1d924da1437dfee4ff243ffb9) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.840 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(52/60) (fad4dc6e3ba4d0be2abbecb7a960c914) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.840 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(53/60) (0ccd9b09fc32f61b45031906a7c612b0) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.840 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(54/60) (1fbc38dc9181a399bd30de4e4044e261) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.840 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(55/60) (2ea15174000a056b0d747d1089f6585f) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.841 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(56/60) (28baca2d42222f0a0182c5c89e1717c6) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.841 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(57/60) (e1abb8bce0113fb035f4a36271b73f17) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.841 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(58/60) (9f4a99aad3d396dbbffc75bc9b40f0b6) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.841 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(59/60) (d58fd6d1d88118d0037c3c2e92176b69) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.841 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt
(60/60) (765e9de291878316b4c69000bcd0b2df) switched from RUNNING to
CANCELING.
2019-10-09 13:54:21.842 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
DeDuplicationBolt (1/30) (782206df8a2ad0f7a2946eedf0cbc827) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.842 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
DeDuplicationBolt (2/30) (db788aa1e9fdb0022bf9d5a7dc257235) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.842 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
DeDuplicationBolt (3/30) (8567c954df44048c35632a52314d9660) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.842 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
DeDuplicationBolt (4/30) (b8d2a0f72c717e2a06e5770751bf20e7) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.843 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
DeDuplicationBolt (5/30) (d72b93dbfa363c38bd6b7451a39c7d67) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.843 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
DeDuplicationBolt (6/30) (559d7f2c0bcb8255df8963b3b4329f66) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.843 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
DeDuplicationBolt (7/30) (8124a163b7727ce53601ba21c7a49acc) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.843 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
DeDuplicationBolt (8/30) (c151f0d09157158e3f449a847c41cc65) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.843 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
DeDuplicationBolt (9/30) (86b2c2178b48d7bb74742036b6489260) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.843 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
DeDuplicationBolt (10/30) (f716beb167fbe51c1db7798c1f1794d3) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.844 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
DeDuplicationBolt (11/30) (b72e2b66c4c3fa24a2e6d247210f9e34) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.844 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
DeDuplicationBolt (12/30) (e63a7ed9a8fb828a1cda0e206fa803e1) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.844 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
DeDuplicationBolt (13/30) (7dcb47fd7e88bdc86ae45344d9779e99) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.844 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
DeDuplicationBolt (14/30) (ccc164979332f39f374664d758345301) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.844 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
DeDuplicationBolt (15/30) (b21287c5c6b9b89124a1cd6810eb2a21) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.844 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
DeDuplicationBolt (16/30) (91071077d5e04266d5b6d9833cf2f9e4) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.845 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
DeDuplicationBolt (17/30) (b5ec2ff0c6decca89ff76223c25b9d9b) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.845 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
DeDuplicationBolt (18/30) (0774604957ec0d2680b8673b98dbbe7b) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.845 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
DeDuplicationBolt (19/30) (6e402ae9f4d0b8755c0c7a2ef67f2c1a) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.845 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
DeDuplicationBolt (20/30) (51493f15a4b7166aa390f854a597c4f6) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.845 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
DeDuplicationBolt (21/30) (e329d85d95a8a6a142b8157bd3b2eeee) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.845 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
DeDuplicationBolt (22/30) (c17a752c676bd4fa1559b6641f959fc7) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.846 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
DeDuplicationBolt (23/30) (94bfdd3b1bb3be163c951b41aa80f5a8) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.846 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
DeDuplicationBolt (24/30) (f4a378189dd403de5d4d5a3e77415f9e) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.846 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
DeDuplicationBolt (25/30) (d16ff3ac286ab1fd32caa9f46fca5619) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.846 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
DeDuplicationBolt (26/30) (090c3aaa6bb23261e74a7bf70060a36d) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.847 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
DeDuplicationBolt (27/30) (9aeebb1f3b6d51be5388674955953c52) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.847 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
DeDuplicationBolt (28/30) (23c2fceab79f540a9d89e2e4f092a865) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.847 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
DeDuplicationBolt (29/30) (e6bb884157821902ae36d036737efaa5) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.847 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
DeDuplicationBolt (30/30) (afca1c29fa6c50a49c88d59d3df459f6) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.847 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
StatisticsBolt (1/30) (9adb75badd9cb8afbb4251a33ef27171) switched from
RUNNING to CANCELING.
2019-10-09 13:54:21.848 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
StatisticsBolt (2/30) (473835002699bfb37c5b88edafd219a1) switched from
RUNNING to CANCELING.
2019-10-09 13:54:21.848 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
StatisticsBolt (3/30) (e7e030cb12de97e633cf473d7e627206) switched from
RUNNING to CANCELING.
2019-10-09 13:54:21.848 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
StatisticsBolt (4/30) (0459e2f2ea079437f62769f2aee861d2) switched from
RUNNING to CANCELING.
2019-10-09 13:54:21.848 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
StatisticsBolt (5/30) (ed428b67cd250fd6407a0b5243aab514) switched from
RUNNING to CANCELING.
2019-10-09 13:54:21.848 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
StatisticsBolt (6/30) (4a54f08765d6c9faac31ace85a406819) switched from
RUNNING to CANCELING.
2019-10-09 13:54:21.849 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
StatisticsBolt (7/30) (24ab50af8fdab9de5f0915479cedc50f) switched from
RUNNING to CANCELING.
2019-10-09 13:54:21.849 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
StatisticsBolt (8/30) (5f365913e8e69f0bb8fad4fcac0337ea) switched from
RUNNING to CANCELING.
2019-10-09 13:54:21.849 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
StatisticsBolt (9/30) (d4ac35d28253d1dd096fa6bab4fafc8a) switched from
RUNNING to CANCELING.
2019-10-09 13:54:21.849 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
StatisticsBolt (10/30) (1ad00f44a299183f81f61ed829410b1d) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.849 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
StatisticsBolt (11/30) (98238896820682b892815dfa2f62e6c2) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.849 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
StatisticsBolt (12/30) (6019139a62f395abb0d3bb0fa51b326e) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.850 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
StatisticsBolt (13/30) (73f72dd701d042dacd95988521873013) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.850 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
StatisticsBolt (14/30) (3109dae5918d772b64e7d8b28b128dfe) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.850 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
StatisticsBolt (15/30) (10c0effec4da40afe144a59556882680) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.850 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
StatisticsBolt (16/30) (722f04d080fed0df95787fc11a7c95d0) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.850 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
StatisticsBolt (17/30) (07461d9203eedf28c0a8c893a631ca2f) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.850 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
StatisticsBolt (18/30) (206c625688c12fa82cfc3bae7b60f24d) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.851 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
StatisticsBolt (19/30) (6fd477a3d452423a09cd0ba0494a6deb) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.851 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
StatisticsBolt (20/30) (d0aa4c46dbb46db09c9395e39b41a4c1) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.851 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
StatisticsBolt (21/30) (0dde7060b76feb5e7333e20d3fe4e7a1) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.851 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
StatisticsBolt (22/30) (ce6749c3e5b996df8647603e4e8fbfdb) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.851 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
StatisticsBolt (23/30) (8270b98412ef6613859f9ae0109850e3) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.851 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
StatisticsBolt (24/30) (136806c81715dd66a3d5cf8f49ceb031) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.852 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
StatisticsBolt (25/30) (e95b22dff10204581a86a35461bf5768) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.852 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
StatisticsBolt (26/30) (9fff9a9c688f9e383f18661923a814ba) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.852 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
StatisticsBolt (27/30) (2626c8dad0fbab32fadd995930211d8a) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.852 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
StatisticsBolt (28/30) (70aa1e618f77758c684cae15b7722752) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.852 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
StatisticsBolt (29/30) (ef3392ed8088af2c05212e0fb0a5aac6) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.853 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  -
StatisticsBolt (30/30) (6af3938acd0f635d95c9c432edeab539) switched
from RUNNING to CANCELING.
2019-10-09 13:54:21.855 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.jobmaster.JobMaster  - Close ResourceManager
connection 490c7cf7fc01baf7e0ece283d48690a4: JobManager is no longer
the leader..
2019-10-09 13:54:21.855 [flink-akka.actor.default-dispatcher-88] INFO
org.apache.flink.runtime.jobmaster.slotpool.SlotPool  - Suspending
SlotPool.
2019-10-09 13:54:21.856 [flink-akka.actor.default-dispatcher-80] INFO
org.apache.flink.runtime.jobmaster.JobMaster  - Stopping the JobMaster
for job hdp-shoujiserver_CommercialDotingAccess(2657b25ff073a91c888ff78a28bd2331).
2019-10-09 13:54:21.856 [flink-akka.actor.default-dispatcher-20] INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcActor  - The rpc endpoint
org.apache.flink.runtime.jobmaster.slotpool.SlotPool has not been
started yet. Discarding message
org.apache.flink.runtime.rpc.messages.LocalRpcInvocation until
processing is started.
2019-10-09 13:54:21.856 [flink-akka.actor.default-dispatcher-20] INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcActor  - The rpc endpoint
org.apache.flink.runtime.jobmaster.slotpool.SlotPool has not been
started yet. Discarding message
org.apache.flink.runtime.rpc.messages.LocalRpcInvocation until
processing is started.
2019-10-09 13:54:21.857 [flink-akka.actor.default-dispatcher-20] INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcActor  - The rpc endpoint
org.apache.flink.runtime.jobmaster.slotpool.SlotPool has not been
started yet. Discarding message
org.apache.flink.runtime.rpc.messages.LocalRpcInvocation until
processing is started.
2019-10-09 13:54:21.857 [flink-akka.actor.default-dispatcher-20] INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcActor  - The rpc endpoint
org.apache.flink.runtime.jobmaster.slotpool.SlotPool has not been
started yet. Discarding message
org.apache.flink.runtime.rpc.messages.LocalRpcInvocation until
processing is started.
2019-10-09 13:54:21.857 [flink-akka.actor.default-dispatcher-20] INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcActor  - The rpc endpoint
org.apache.flink.runtime.jobmaster.slotpool.SlotPool has not been
started yet. Discarding message
org.apache.flink.runtime.rpc.messages.LocalRpcInvocation until
processing is started.
2019-10-09 13:54:21.857 [flink-akka.actor.default-dispatcher-20] INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcActor  - The rpc endpoint
org.apache.flink.runtime.jobmaster.slotpool.SlotPool has not been
started yet. Discarding message
org.apache.flink.runtime.rpc.messages.LocalRpcInvocation until
processing is started.
2019-10-09 13:54:21.857 [flink-akka.actor.default-dispatcher-20] INFO
org.apache.flink.runtime.jobmaster.slotpool.SlotPool  - Stopping
SlotPool.
2019-10-09 13:54:21.857 [flink-akka.actor.default-dispatcher-20] INFO
o.a.f.runtime.leaderelection.ZooKeeperLeaderElectionService  -
Stopping ZooKeeperLeaderElectionService
ZooKeeperLeaderElectionService{leaderPath='/leader/2657b25ff073a91c888ff78a28bd2331/job_manager_lock'}.
2019-10-09 13:54:22.602 [main-SendThread(10.160.101.29:2181)] WARN
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL
configuration failed: javax.security.auth.login.LoginException: No
JAAS configuration section named 'Client' was found in specified JAAS
configuration file: '/tmp/jaas-4748011049181233444.conf'. Will
continue connection to Zookeeper server without SASL authentication,
if Zookeeper server allows it.
2019-10-09 13:54:22.603 [main-SendThread(10.160.101.29:2181)] INFO
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Opening
socket connection to server 10.160.101.29/10.160.101.29:2181
2019-10-09 13:54:22.603 [main-EventThread] ERROR
o.a.flink.shaded.curator.org.apache.curator.ConnectionState  -
Authentication failed
2019-10-09 13:54:22.603 [main-SendThread(10.160.101.29:2181)] WARN
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Session
0x4597758a3589ca6 for server null, unexpected error, closing socket
connection and attempting reconnect
java.net.ConnectException: 拒绝连接
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
	at org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
2019-10-09 13:54:22.921 [main-SendThread(10.160.101.28:2181)] WARN
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL
configuration failed: javax.security.auth.login.LoginException: No
JAAS configuration section named 'Client' was found in specified JAAS
configuration file: '/tmp/jaas-4748011049181233444.conf'. Will
continue connection to Zookeeper server without SASL authentication,
if Zookeeper server allows it.
2019-10-09 13:54:22.922 [main-SendThread(10.160.101.28:2181)] INFO
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Opening
socket connection to server 10.160.101.28/10.160.101.28:2181
2019-10-09 13:54:22.922 [main-EventThread] ERROR
o.a.flink.shaded.curator.org.apache.curator.ConnectionState  -
Authentication failed
2019-10-09 13:54:22.922 [main-SendThread(10.160.101.28:2181)] WARN
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Session
0x4597758a3589ca6 for server null, unexpected error, closing socket
connection and attempting reconnect
java.net.ConnectException: 拒绝连接
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
	at org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
2019-10-09 13:54:24.008 [main-SendThread(10.160.101.26:2181)] WARN
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL
configuration failed: javax.security.auth.login.LoginException: No
JAAS configuration section named 'Client' was found in specified JAAS
configuration file: '/tmp/jaas-4748011049181233444.conf'. Will
continue connection to Zookeeper server without SASL authentication,
if Zookeeper server allows it.
2019-10-09 13:54:24.008 [main-SendThread(10.160.101.26:2181)] INFO
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Opening
socket connection to server 10.160.101.26/10.160.101.26:2181
2019-10-09 13:54:24.008 [main-EventThread] ERROR
o.a.flink.shaded.curator.org.apache.curator.ConnectionState  -
Authentication failed
2019-10-09 13:54:24.009 [main-SendThread(10.160.101.26:2181)] WARN
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Session
0x4597758a3589ca6 for server null, unexpected error, closing socket
connection and attempting reconnect
java.net.ConnectException: 拒绝连接
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
	at org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
2019-10-09 13:54:24.778 [main-SendThread(10.160.101.30:2181)] WARN
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL
configuration failed: javax.security.auth.login.LoginException: No
JAAS configuration section named 'Client' was found in specified JAAS
configuration file: '/tmp/jaas-4748011049181233444.conf'. Will
continue connection to Zookeeper server without SASL authentication,
if Zookeeper server allows it.
2019-10-09 13:54:24.778 [main-SendThread(10.160.101.30:2181)] INFO
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Opening
socket connection to server 10.160.101.30/10.160.101.30:2181
2019-10-09 13:54:24.778 [main-EventThread] ERROR
o.a.flink.shaded.curator.org.apache.curator.ConnectionState  -
Authentication failed
2019-10-09 13:54:24.779 [main-SendThread(10.160.101.30:2181)] WARN
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Session
0x4597758a3589ca6 for server null, unexpected error, closing socket
connection and attempting reconnect
java.net.ConnectException: 拒绝连接
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
	at org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
2019-10-09 13:54:26.748 [main-SendThread(10.160.101.27:2181)] WARN
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL
configuration failed: javax.security.auth.login.LoginException: No
JAAS configuration section named 'Client' was found in specified JAAS
configuration file: '/tmp/jaas-4748011049181233444.conf'. Will
continue connection to Zookeeper server without SASL authentication,
if Zookeeper server allows it.
2019-10-09 13:54:26.748 [main-SendThread(10.160.101.27:2181)] INFO
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Opening
socket connection to server 10.160.101.27/10.160.101.27:2181
2019-10-09 13:54:26.749 [main-EventThread] ERROR
o.a.flink.shaded.curator.org.apache.curator.ConnectionState  -
Authentication failed
2019-10-09 13:54:26.749 [main-SendThread(10.160.101.27:2181)] INFO
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Socket
connection established to 10.160.101.27/10.160.101.27:2181, initiating
session
2019-10-09 13:54:26.751 [main-SendThread(10.160.101.27:2181)] INFO
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Unable
to read additional data from server sessionid 0x4597758a3589ca6,
likely server has closed socket, closing socket connection and
attempting reconnect
2019-10-09 13:54:27.673 [main-SendThread(10.160.101.29:2181)] WARN
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL
configuration failed: javax.security.auth.login.LoginException: No
JAAS configuration section named 'Client' was found in specified JAAS
configuration file: '/tmp/jaas-4748011049181233444.conf'. Will
continue connection to Zookeeper server without SASL authentication,
if Zookeeper server allows it.
2019-10-09 13:54:27.674 [main-SendThread(10.160.101.29:2181)] INFO
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Opening
socket connection to server 10.160.101.29/10.160.101.29:2181
2019-10-09 13:54:27.674 [main-EventThread] ERROR
o.a.flink.shaded.curator.org.apache.curator.ConnectionState  -
Authentication failed
2019-10-09 13:54:27.674 [main-SendThread(10.160.101.29:2181)] INFO
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Socket
connection established to 10.160.101.29/10.160.101.29:2181, initiating
session
2019-10-09 13:54:27.675 [main-SendThread(10.160.101.29:2181)] INFO
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Unable
to read additional data from server sessionid 0x4597758a3589ca6,
likely server has closed socket, closing socket connection and
attempting reconnect
2019-10-09 13:54:27.865 [main-SendThread(10.160.101.28:2181)] WARN
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL
configuration failed: javax.security.auth.login.LoginException: No
JAAS configuration section named 'Client' was found in specified JAAS
configuration file: '/tmp/jaas-4748011049181233444.conf'. Will
continue connection to Zookeeper server without SASL authentication,
if Zookeeper server allows it.
2019-10-09 13:54:27.865 [main-SendThread(10.160.101.28:2181)] INFO
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Opening
socket connection to server 10.160.101.28/10.160.101.28:2181
2019-10-09 13:54:27.865 [main-EventThread] ERROR
o.a.flink.shaded.curator.org.apache.curator.ConnectionState  -
Authentication failed
2019-10-09 13:54:27.866 [main-SendThread(10.160.101.28:2181)] INFO
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Socket
connection established to 10.160.101.28/10.160.101.28:2181, initiating
session
2019-10-09 13:54:27.867 [main-SendThread(10.160.101.28:2181)] INFO
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Unable
to read additional data from server sessionid 0x4597758a3589ca6,
likely server has closed socket, closing socket connection and
attempting reconnect
2019-10-09 13:54:28.078 [main-SendThread(10.160.101.26:2181)] WARN
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL
configuration failed: javax.security.auth.login.LoginException: No
JAAS configuration section named 'Client' was found in specified JAAS
configuration file: '/tmp/jaas-4748011049181233444.conf'. Will
continue connection to Zookeeper server without SASL authentication,
if Zookeeper server allows it.
2019-10-09 13:54:28.078 [main-SendThread(10.160.101.26:2181)] INFO
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Opening
socket connection to server 10.160.101.26/10.160.101.26:2181
2019-10-09 13:54:28.079 [main-EventThread] ERROR
o.a.flink.shaded.curator.org.apache.curator.ConnectionState  -
Authentication failed
2019-10-09 13:54:28.079 [main-SendThread(10.160.101.26:2181)] INFO
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Socket
connection established to 10.160.101.26/10.160.101.26:2181, initiating
session
2019-10-09 13:54:28.080 [main-SendThread(10.160.101.26:2181)] INFO
o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Session
establishment complete on server 10.160.101.26/10.160.101.26:2181,
sessionid = 0x4597758a3589ca6, negotiated timeout = 40000
2019-10-09 13:54:28.081 [main-EventThread] INFO
o.a.f.s.c.o.a.curator.framework.state.ConnectionStateManager  - State
change: RECONNECTED
2019-10-09 13:54:28.081 [Curator-ConnectionStateManager-0] INFO
o.a.f.runtime.leaderelection.ZooKeeperLeaderElectionService  -
Connection to ZooKeeper was reconnected. Leader election can be
restarted.
2019-10-09 13:54:28.082 [Curator-PathChildrenCache-0] INFO
o.a.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore  -
ZooKeeper connection RECONNECTED. Changes to the submitted job graphs
are monitored again.
2019-10-09 13:54:28.082 [Curator-ConnectionStateManager-0] INFO
o.a.f.r.leaderretrieval.ZooKeeperLeaderRetrievalService  - Connection
to ZooKeeper was reconnected. Leader retrieval can be restarted.
2019-10-09 13:54:28.082 [Curator-ConnectionStateManager-0] INFO
o.a.f.runtime.leaderelection.ZooKeeperLeaderElectionService  -
Connection to ZooKeeper was reconnected. Leader election can be
restarted.
2019-10-09 13:54:28.083 [Curator-ConnectionStateManager-0] INFO
o.a.f.r.leaderretrieval.ZooKeeperLeaderRetrievalService  - Connection
to ZooKeeper was reconnected. Leader retrieval can be restarted.
2019-10-09 13:54:28.083 [Curator-ConnectionStateManager-0] INFO
o.a.f.runtime.leaderelection.ZooKeeperLeaderElectionService  -
Connection to ZooKeeper was reconnected. Leader election can be
restarted.
2019-10-09 13:54:28.087 [flink-akka.actor.default-dispatcher-88] INFO
o.a.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore  -
Released locks of job graph 2657b25ff073a91c888ff78a28bd2331 from
ZooKeeper.
2019-10-09 13:54:28.088 [main-EventThread] INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint  -
http://10.160.140.216:52303 was granted leadership with
leaderSessionID=d80d351c-56ca-490f-aa78-681847d58cfe
2019-10-09 13:54:28.092 [flink-akka.actor.default-dispatcher-83] INFO
org.apache.flink.yarn.YarnResourceManager  - ResourceManager
akka.tcp://flink@10.160.140.216:26073/user/resourcemanager was granted
leadership with fencing token 8b328120280016029994e898e40749f8
2019-10-09 13:54:28.092 [flink-akka.actor.default-dispatcher-83] INFO
o.a.flink.runtime.resourcemanager.slotmanager.SlotManager  - Starting
the SlotManager.
2019-10-09 13:54:28.092 [flink-akka.actor.default-dispatcher-48] INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher  - Dispatcher
akka.tcp://flink@10.160.140.216:26073/user/dispatcher was granted
leadership with fencing token 65d868d3-0665-4555-ae75-86975bb3c7a9
2019-10-09 13:54:28.093 [flink-akka.actor.default-dispatcher-72] INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher  - Recovering
all persisted jobs.
2019-10-09 13:54:28.130 [flink-akka.actor.default-dispatcher-72] INFO
o.a.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore  -
Recovered SubmittedJobGraph(2657b25ff073a91c888ff78a28bd2331, null).
2019-10-09 13:54:28.212 [flink-akka.actor.default-dispatcher-72] INFO
org.apache.flink.yarn.YarnResourceManager  - Registering TaskManager
with ResourceID container_e64_1564717237973_1572220_01_000003
(akka.tcp://flink@10.160.167.236:37058/user/taskmanager_0) at
ResourceManager
2019-10-09 13:54:28.515 [flink-akka.actor.default-dispatcher-72] INFO
org.apache.flink.yarn.YarnResourceManager  - Registering TaskManager
with ResourceID container_e64_1564717237973_1572220_01_000005
(akka.tcp://flink@10.162.92.119:34318/user/taskmanager_0) at
ResourceManager
2019-10-09 13:54:29.072 [flink-akka.actor.default-dispatcher-72] INFO
org.apache.flink.yarn.YarnResourceManager  - Registering TaskManager
with ResourceID container_e64_1564717237973_1572220_01_000008
(akka.tcp://flink@10.160.132.173:48920/user/taskmanager_0) at
ResourceManager
2019-10-09 13:54:29.473 [flink-akka.actor.default-dispatcher-88] INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcService  - Starting RPC
endpoint for org.apache.flink.runtime.jobmaster.JobMaster at
akka://flink/user/jobmanager_3 .
2019-10-09 13:54:29.474 [flink-akka.actor.default-dispatcher-88] INFO
org.apache.flink.runtime.jobmaster.JobMaster  - Initializing job
hdp-shoujiserver_CommercialDotingAccess
(2657b25ff073a91c888ff78a28bd2331).
2019-10-09 13:54:29.487 [flink-akka.actor.default-dispatcher-88] INFO
org.apache.flink.runtime.jobmaster.JobMaster  - Using restart strategy
FailureRateRestartStrategy(failuresInterval=3600000
msdelayInterval=30000 msmaxFailuresPerInterval=3) for
hdp-shoujiserver_CommercialDotingAccess
(2657b25ff073a91c888ff78a28bd2331).
2019-10-09 13:54:29.489 [flink-akka.actor.default-dispatcher-88] INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcService  - Starting RPC
endpoint for org.apache.flink.runtime.jobmaster.slotpool.SlotPool at
akka://flink/user/25b2e77f-fe0c-4e2d-af38-8900a36077d7 .
2019-10-09 13:54:29.489 [flink-akka.actor.default-dispatcher-88] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - Job recovers
via failover strategy: full graph restart
2019-10-09 13:54:29.490 [flink-akka.actor.default-dispatcher-88] INFO
org.apache.flink.runtime.jobmaster.JobMaster  - Running initialization
on master for job hdp-shoujiserver_CommercialDotingAccess
(2657b25ff073a91c888ff78a28bd2331).
2019-10-09 13:54:29.490 [flink-akka.actor.default-dispatcher-88] INFO
org.apache.flink.runtime.jobmaster.JobMaster  - Successfully ran
initialization on master in 0 ms.
2019-10-09 13:54:29.497 [flink-akka.actor.default-dispatcher-88] INFO
o.a.f.runtime.checkpoint.ZooKeeperCompletedCheckpointStore  -
Initialized in '/checkpoints/2657b25ff073a91c888ff78a28bd2331'.
2019-10-09 13:54:29.498 [flink-akka.actor.default-dispatcher-88] INFO
org.apache.flink.runtime.jobmaster.JobMaster  - Loading state backend
via factory org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory
2019-10-09 13:54:29.498 [flink-akka.actor.default-dispatcher-88] WARN
org.apache.flink.configuration.Configuration  - Config uses deprecated
configuration key 'state.backend.rocksdb.checkpointdir' instead of
proper key 'state.backend.rocksdb.localdir'
2019-10-09 13:54:29.502 [flink-akka.actor.default-dispatcher-88] INFO
o.a.f.runtime.checkpoint.ZooKeeperCompletedCheckpointStore  -
Recovering checkpoints from ZooKeeper.
2019-10-09 13:54:29.513 [flink-akka.actor.default-dispatcher-88] INFO
o.a.f.runtime.checkpoint.ZooKeeperCompletedCheckpointStore  - Found 5
checkpoints in ZooKeeper.
2019-10-09 13:54:29.514 [flink-akka.actor.default-dispatcher-88] INFO
o.a.f.runtime.checkpoint.ZooKeeperCompletedCheckpointStore  - Trying
to fetch 5 checkpoints from storage.
2019-10-09 13:54:29.514 [flink-akka.actor.default-dispatcher-88] INFO
o.a.f.runtime.checkpoint.ZooKeeperCompletedCheckpointStore  - Trying
to retrieve checkpoint 33036.
2019-10-09 13:54:29.516 [flink-akka.actor.default-dispatcher-21] INFO
org.apache.flink.yarn.YarnResourceManager  - Registering TaskManager
with ResourceID container_e64_1564717237973_1572220_01_000010
(akka.tcp://flink@10.160.124.173:30267/user/taskmanager_0) at
ResourceManager
2019-10-09 13:54:29.523 [flink-akka.actor.default-dispatcher-88] INFO
o.a.f.runtime.checkpoint.ZooKeeperCompletedCheckpointStore  - Trying
to retrieve checkpoint 33037.
2019-10-09 13:54:29.532 [flink-akka.actor.default-dispatcher-88] INFO
o.a.f.runtime.checkpoint.ZooKeeperCompletedCheckpointStore  - Trying
to retrieve checkpoint 33038.
2019-10-09 13:54:29.540 [flink-akka.actor.default-dispatcher-88] INFO
o.a.f.runtime.checkpoint.ZooKeeperCompletedCheckpointStore  - Trying
to retrieve checkpoint 33039.
2019-10-09 13:54:29.549 [flink-akka.actor.default-dispatcher-88] INFO
o.a.f.runtime.checkpoint.ZooKeeperCompletedCheckpointStore  - Trying
to retrieve checkpoint 33040.
2019-10-09 13:54:30.712 [flink-akka.actor.default-dispatcher-88] INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator  - Restoring
job 2657b25ff073a91c888ff78a28bd2331 from latest valid checkpoint:
Checkpoint 33040 @ 1570600391503 for 2657b25ff073a91c888ff78a28bd2331.
2019-10-09 13:54:30.715 [flink-akka.actor.default-dispatcher-88] INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator  - No master
state to restore
2019-10-09 13:54:30.715 [flink-akka.actor.default-dispatcher-88] INFO
o.a.f.runtime.leaderelection.ZooKeeperLeaderElectionService  -
Starting ZooKeeperLeaderElectionService
ZooKeeperLeaderElectionService{leaderPath='/leader/2657b25ff073a91c888ff78a28bd2331/job_manager_lock'}.
2019-10-09 13:54:30.721 [main-EventThread] INFO
org.apache.flink.runtime.jobmaster.JobManagerRunner  - JobManager
runner for job hdp-shoujiserver_CommercialDotingAccess
(2657b25ff073a91c888ff78a28bd2331) was granted leadership with session
id 829a1a30-be5a-4e4a-aa0c-21cc71b45599 at
akka.tcp://flink@10.160.140.216:26073/user/jobmanager_3.
2019-10-09 13:54:30.724 [flink-akka.actor.default-dispatcher-21] INFO
o.a.f.r.leaderretrieval.ZooKeeperLeaderRetrievalService  - Starting
ZooKeeperLeaderRetrievalService /leader/resource_manager_lock.
2019-10-09 13:54:30.724 [flink-akka.actor.default-dispatcher-21] INFO
org.apache.flink.runtime.jobmaster.JobMaster  - Starting execution of
job hdp-shoujiserver_CommercialDotingAccess
(2657b25ff073a91c888ff78a28bd2331)
2019-10-09 13:54:30.725 [flink-akka.actor.default-dispatcher-21] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - Job
hdp-shoujiserver_CommercialDotingAccess
(2657b25ff073a91c888ff78a28bd2331) switched from state CREATED to
RUNNING.
2019-10-09 13:54:30.725 [flink-akka.actor.default-dispatcher-21] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph  - Source:
WeiShiDataAnalysisSpout (1/8) (d072a1bc3f3dbf7896ef619951d91c88)
switched from CREATED to SCHEDULED.

*After the zk reconnected after a suspend, the job was recovered, and
you can see, there was a StandaloneDispatcher created.
*



*Don't know why the 'FencingTokenException: Fencing token not set'
exception happened, and don't know why different dispatchers were
created.*


*Thanks*


*Joshua*



On Wed, Oct 2, 2019 at 9:10 PM Till Rohrmann <trohrmann@apache.org> wrote:

> Hi Bruce, are you able to provide us with the full debug logs? From the
> excerpt itself it is hard to tell what is going on.
>
> Cheers,
> Till
>
> On Wed, Oct 2, 2019 at 2:24 PM Fabian Hueske <fhueske@gmail.com> wrote:
>
>> Hi Bruce,
>>
>> I haven't seen such an exception yet, but maybe Till (in CC) can help.
>>
>> Best,
>> Fabian
>>
>> Am Di., 1. Okt. 2019 um 05:51 Uhr schrieb Hanson, Bruce <
>> bruce.hanson@here.com>:
>>
>>> Hi all,
>>>
>>>
>>>
>>> We are running some of our Flink jobs with Job Manager High
>>> Availability. Occasionally we get a cluster that comes up improperly and
>>> doesn’t respond. Attempts to submit the job seem to hang and when we hit
>>> the /overview REST endpoint in the Job Manager we get a 500 error and a
>>> fencing token exception like this:
>>>
>>>
>>>
>>> *2019-09-21 05:04:07.785 [flink-akka.actor.default-dispatcher-4428]
>>> level=ERROR o.a.f.runtime.rest.handler.cluster.ClusterOverviewHandler  -
>>> Implementation error: Unhandled exception.*
>>>
>>> *org.apache.flink.runtime.rpc.exceptions.FencingTokenException: Fencing
>>> token not set: Ignoring message LocalFencedMessage(null,
>>> LocalRpcInvocation(requestResourceOverview(Time))) sent to
>>> akka.tcp://flink@job-ef80a156-3350-4e85-8761-b0e42edc346f-jm-0.job-ef80a156-3350-4e85-8761-b0e42edc346f-jm-svc.olp-here-test-j-ef80a156-3350-4e85-8761-b0e42edc346f.svc.cluster.local:6126/user/resourcemanager
>>> because the fencing token is null.*
>>>
>>> *        at
>>> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:59)*
>>>
>>> *        at
>>> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:142)*
>>>
>>> *        at
>>> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.onReceive(FencedAkkaRpcActor.java:40)*
>>>
>>> *        at
>>> akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)*
>>>
>>> *        at akka.actor.Actor$class.aroundReceive(Actor.scala:502)*
>>>
>>> *        at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)*
>>>
>>> *        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)*
>>>
>>> *        at akka.actor.ActorCell.invoke(ActorCell.scala:495)*
>>>
>>> *        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)*
>>>
>>> *        at akka.dispatch.Mailbox.run(Mailbox.scala:224)*
>>>
>>> *        at akka.dispatch.Mailbox.exec(Mailbox.scala:234)*
>>>
>>> *        at
>>> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)*
>>>
>>> *        at
>>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)*
>>>
>>> *        at
>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)*
>>>
>>> *        at
>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)*
>>>
>>>
>>>
>>>
>>>
>>> We are running Flink 1.7.1 in Kubernetes and run each job in its own
>>> namespace with a three-node Zookeeper cluster and two Job Managers, plus
>>> one or more Task Managers. I have been able to replicate the issue, but
>>> don’t find any difference in the logs between a failing cluster and a good
>>> one.
>>>
>>>
>>>
>>> Does anyone here have any ideas as to what’s happening, or what I should
>>> be looking for?
>>>
>>>
>>>
>>> -Bruce
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> [image: cid:image001.png@01D2B473.0F7F85E0]
>>>
>>>
>>>
>>> *Bruce Hanson*
>>>
>>> *Principal Engineer*
>>>
>>> *M: +1 425 681 0422*
>>>
>>>
>>>
>>> HERE Technologies
>>>
>>> 701 Pike Street, Suite 2000
>>>
>>> Seattle, WA 98101 USA
>>>
>>> *47° 36' 41" N 122° 19' 57" W*
>>>
>>>
>>>
>>> [image: cid:image002.png@01D2B473.0F7F85E0] <http://360.here.com/>    [image:
>>> cid:image003.png@01D2B473.0F7F85E0] <https://www.twitter.com/here>   [image:
>>> cid:image004.png@01D2B473.0F7F85E0] <https://www.facebook.com/here>    [image:
>>> cid:image005.png@01D2B473.0F7F85E0]
>>> <https://www.linkedin.com/company/heremaps>    [image:
>>> cid:image006.png@01D2B473.0F7F85E0] <https://www.instagram.com/here/>
>>>
>>>
>>>
>>>
>>>
>>

Mime
View raw message