Hi Till

Recently, this issue happened frequently, but some jobs recovered from the exception, some not.

for the job who can not recover from the exception, the log pasted below:

2019-10-09 13:54:21.746 [main-EventThread] INFO  o.a.f.s.c.o.a.curator.framework.state.ConnectionStateManager  - State change: SUSPENDED
2019-10-09 13:54:21.746 [Curator-ConnectionStateManager-0] WARN  o.a.f.runtime.leaderelection.ZooKeeperLeaderElectionService  - Connection to ZooKeeper suspended. The contender akka.tcp://flink@10.162.89.153:16736/user/dispatcher no longer participates in the leader election.
2019-10-09 13:54:21.746 [flink-akka.actor.default-dispatcher-46] INFO  org.apache.flink.yarn.YarnResourceManager  - ResourceManager akka.tcp://flink@10.162.89.153:16736/user/resourcemanager was revoked leadership. Clearing fencing token.
2019-10-09 13:54:21.746 [Curator-ConnectionStateManager-0] WARN  o.a.f.runtime.leaderelection.ZooKeeperLeaderElectionService  - Connection to ZooKeeper suspended. The contender http://10.162.89.153:12503 no longer participates in the leader election.
2019-10-09 13:54:21.747 [Curator-ConnectionStateManager-0] INFO  o.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint  - http://10.162.89.153:12503 lost leadership
2019-10-09 13:54:21.747 [Curator-ConnectionStateManager-0] WARN  o.a.f.runtime.leaderelection.ZooKeeperLeaderElectionService  - Connection to ZooKeeper suspended. The contender akka.tcp://flink@10.162.89.153:16736/user/resourcemanager no longer participates in the leader election.
2019-10-09 13:54:21.747 [Curator-ConnectionStateManager-0] WARN  o.a.f.r.leaderretrieval.ZooKeeperLeaderRetrievalService  - Connection to ZooKeeper suspended. Can no longer retrieve the leader from ZooKeeper.
2019-10-09 13:54:21.747 [Curator-ConnectionStateManager-0] WARN  o.a.f.r.leaderretrieval.ZooKeeperLeaderRetrievalService  - Connection to ZooKeeper suspended. Can no longer retrieve the leader from ZooKeeper.
2019-10-09 13:54:21.747 [flink-akka.actor.default-dispatcher-43] INFO  org.apache.flink.runtime.dispatcher.MiniDispatcher  - Dispatcher akka.tcp://flink@10.162.89.153:16736/user/dispatcher was revoked leadership.
2019-10-09 13:54:21.747 [flink-akka.actor.default-dispatcher-46] INFO  o.a.flink.runtime.resourcemanager.slotmanager.SlotManager  - Suspending the SlotManager.
2019-10-09 13:54:22.072 [main-SendThread(10.160.101.26:2181)] WARN  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/tmp/jaas-1334102078344377682.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2019-10-09 13:54:22.072 [main-SendThread(10.160.101.26:2181)] INFO  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Opening socket connection to server 10.160.101.26/10.160.101.26:2181
2019-10-09 13:54:22.072 [main-EventThread] ERROR o.a.flink.shaded.curator.org.apache.curator.ConnectionState  - Authentication failed
2019-10-09 13:54:22.072 [main-SendThread(10.160.101.26:2181)] WARN  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Session 0x2597758ae5091ff for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
	at org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
2019-10-09 13:54:22.973 [main-SendThread(10.160.101.27:2181)] WARN  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/tmp/jaas-1334102078344377682.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2019-10-09 13:54:22.973 [main-SendThread(10.160.101.27:2181)] INFO  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Opening socket connection to server 10.160.101.27/10.160.101.27:2181
2019-10-09 13:54:22.973 [main-EventThread] ERROR o.a.flink.shaded.curator.org.apache.curator.ConnectionState  - Authentication failed
2019-10-09 13:54:22.974 [main-SendThread(10.160.101.27:2181)] WARN  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Session 0x2597758ae5091ff for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
	at org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
2019-10-09 13:54:23.405 [main-SendThread(10.160.101.30:2181)] WARN  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/tmp/jaas-1334102078344377682.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2019-10-09 13:54:23.405 [main-SendThread(10.160.101.30:2181)] INFO  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Opening socket connection to server 10.160.101.30/10.160.101.30:2181
2019-10-09 13:54:23.405 [main-EventThread] ERROR o.a.flink.shaded.curator.org.apache.curator.ConnectionState  - Authentication failed
2019-10-09 13:54:23.406 [main-SendThread(10.160.101.30:2181)] WARN  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Session 0x2597758ae5091ff for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
	at org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
2019-10-09 13:54:24.379 [main-SendThread(10.160.101.28:2181)] WARN  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/tmp/jaas-1334102078344377682.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2019-10-09 13:54:24.379 [main-SendThread(10.160.101.28:2181)] INFO  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Opening socket connection to server 10.160.101.28/10.160.101.28:2181
2019-10-09 13:54:24.379 [main-EventThread] ERROR o.a.flink.shaded.curator.org.apache.curator.ConnectionState  - Authentication failed
2019-10-09 13:54:24.380 [main-SendThread(10.160.101.28:2181)] WARN  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Session 0x2597758ae5091ff for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
	at org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
2019-10-09 13:54:26.044 [main-SendThread(10.160.101.29:2181)] WARN  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/tmp/jaas-1334102078344377682.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2019-10-09 13:54:26.045 [main-SendThread(10.160.101.29:2181)] INFO  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Opening socket connection to server 10.160.101.29/10.160.101.29:2181
2019-10-09 13:54:26.045 [main-EventThread] ERROR o.a.flink.shaded.curator.org.apache.curator.ConnectionState  - Authentication failed
2019-10-09 13:54:29.050 [main-SendThread(10.160.101.29:2181)] INFO  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Socket connection established to 10.160.101.29/10.160.101.29:2181, initiating session
2019-10-09 13:54:29.095 [main-SendThread(10.160.101.29:2181)] INFO  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Session establishment complete on server 10.160.101.29/10.160.101.29:2181, sessionid = 0x2597758ae5091ff, negotiated timeout = 40000
2019-10-09 13:54:29.095 [main-EventThread] INFO  o.a.f.s.c.o.a.curator.framework.state.ConnectionStateManager  - State change: RECONNECTED
2019-10-09 13:54:29.096 [Curator-ConnectionStateManager-0] INFO  o.a.f.runtime.leaderelection.ZooKeeperLeaderElectionService  - Connection to ZooKeeper was reconnected. Leader election can be restarted.
2019-10-09 13:54:29.097 [Curator-ConnectionStateManager-0] INFO  o.a.f.runtime.leaderelection.ZooKeeperLeaderElectionService  - Connection to ZooKeeper was reconnected. Leader election can be restarted.
2019-10-09 13:54:29.097 [Curator-ConnectionStateManager-0] INFO  o.a.f.runtime.leaderelection.ZooKeeperLeaderElectionService  - Connection to ZooKeeper was reconnected. Leader election can be restarted.
2019-10-09 13:54:29.097 [Curator-ConnectionStateManager-0] INFO  o.a.f.r.leaderretrieval.ZooKeeperLeaderRetrievalService  - Connection to ZooKeeper was reconnected. Leader retrieval can be restarted.
2019-10-09 13:54:29.097 [Curator-ConnectionStateManager-0] INFO  o.a.f.r.leaderretrieval.ZooKeeperLeaderRetrievalService  - Connection to ZooKeeper was reconnected. Leader retrieval can be restarted.
2019-10-09 13:54:29.105 [flink-akka.actor.default-dispatcher-46] INFO  org.apache.flink.yarn.YarnResourceManager  - ResourceManager akka.tcp://flink@10.162.89.153:16736/user/resourcemanager was granted leadership with fencing token a22e4c9c80fd0c7249102f69c0b64e01
2019-10-09 13:54:29.105 [flink-akka.actor.default-dispatcher-46] INFO  o.a.flink.runtime.resourcemanager.slotmanager.SlotManager  - Starting the SlotManager.
2019-10-09 13:54:29.107 [main-EventThread] INFO  o.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint  - http://10.162.89.153:12503 was granted leadership with leaderSessionID=53b02e00-95c3-4d34-a272-b84795b991a7
2019-10-09 13:54:29.116 [flink-akka.actor.default-dispatcher-26] INFO  org.apache.flink.runtime.dispatcher.MiniDispatcher  - Dispatcher akka.tcp://flink@10.162.89.153:16736/user/dispatcher was granted leadership with fencing token d645a3fe-8fee-453b-abd5-bd4cd82ebba1
2019-10-09 13:54:34.809 [flink-rest-server-netty-worker-thread-73] ERROR org.apache.flink.runtime.rest.handler.job.JobDetailsHandler  - Could not retrieve the redirect address.
java.util.concurrent.CompletionException: org.apache.flink.runtime.rpc.exceptions.FencingTokenException: Fencing token not set: Ignoring message LocalFencedMessage(9f126af7b71d8b61dcb3687f3dd84166, LocalRpcInvocation(requestRestAddress(Time))) sent to akka.tcp://flink@10.162.89.153:16736/user/dispatcher because the fencing token is null.
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
	at org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:772)
	at akka.dispatch.OnComplete.internal(Future.scala:258)
	at akka.dispatch.OnComplete.internal(Future.scala:256)
	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)
	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
	at org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:83)
	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
	at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:534)
	at akka.actor.ActorRef.tell(ActorRef.scala:130)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.sendErrorIfSender(AkkaRpcActor.java:371)
	at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:57)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:142)
	at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.onReceive(FencedAkkaRpcActor.java:40)
	at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
	at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
	at akka.actor.ActorCell.invoke(ActorCell.scala:495)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
	at akka.dispatch.Mailbox.run(Mailbox.scala:224)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.apache.flink.runtime.rpc.exceptions.FencingTokenException: Fencing token not set: Ignoring message LocalFencedMessage(9f126af7b71d8b61dcb3687f3dd84166, LocalRpcInvocation(requestRestAddress(Time))) sent to akka.tcp://flink@10.162.89.153:16736/user/dispatcher because the fencing token is null.
	at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:59)
	... 14 common frames omitted
2019-10-09 13:54:56.007 [flink-rest-server-netty-worker-thread-74] ERROR org.apache.flink.runtime.rest.handler.job.JobDetailsHandler  - Could not retrieve the redirect address.
java.util.concurrent.CompletionException: org.apache.flink.runtime.rpc.exceptions.FencingTokenException: Fencing token not set: Ignoring message LocalFencedMessage(9f126af7b71d8b61dcb3687f3dd84166, LocalRpcInvocation(requestRestAddress(Time))) sent to akka.tcp://flink@10.162.89.153:16736/user/dispatcher because the fencing token is null.
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
	at org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:772)
	at akka.dispatch.OnComplete.internal(Future.scala:258)
	at akka.dispatch.OnComplete.internal(Future.scala:256)
	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)
	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
	at org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:83)
	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
	at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:534)
	at akka.actor.ActorRef.tell(ActorRef.scala:130)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.sendErrorIfSender(AkkaRpcActor.java:371)
	at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:57)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:142)
	at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.onReceive(FencedAkkaRpcActor.java:40)
	at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
	at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
	at akka.actor.ActorCell.invoke(ActorCell.scala:495)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
	at akka.dispatch.Mailbox.run(Mailbox.scala:224)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.apache.flink.runtime.rpc.exceptions.FencingTokenException: Fencing token not set: Ignoring message LocalFencedMessage(9f126af7b71d8b61dcb3687f3dd84166, LocalRpcInvocation(requestRestAddress(Time))) sent to akka.tcp://flink@10.162.89.153:16736/user/dispatcher because the fencing token is null.
	at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:59)
	... 14 common frames omitted

In the above log, the ha zk got reconnected after a suspend, but the job can not recover, and there existed a MiniDispatcherRestEndpoint.


for the job who can recover from the same condition, the log pasted as below:

2019-10-09 13:54:21.803 [main-EventThread] INFO  o.a.f.s.c.o.a.curator.framework.state.ConnectionStateManager  - State change: SUSPENDED
2019-10-09 13:54:21.804 [Curator-ConnectionStateManager-0] WARN  o.a.f.runtime.leaderelection.ZooKeeperLeaderElectionService  - Connection to ZooKeeper suspended. The contender http://10.160.140.216:52303 no longer participates in the leader election.
2019-10-09 13:54:21.804 [Curator-ConnectionStateManager-0] INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint  - http://10.160.140.216:52303 lost leadership
2019-10-09 13:54:21.804 [Curator-PathChildrenCache-0] WARN  o.a.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore  - ZooKeeper connection SUSPENDING. Changes to the submitted job graphs are not monitored (temporarily).
2019-10-09 13:54:21.805 [Curator-ConnectionStateManager-0] WARN  o.a.f.r.leaderretrieval.ZooKeeperLeaderRetrievalService  - Connection to ZooKeeper suspended. Can no longer retrieve the leader from ZooKeeper.
2019-10-09 13:54:21.805 [flink-akka.actor.default-dispatcher-88] INFO  org.apache.flink.yarn.YarnResourceManager  - ResourceManager akka.tcp://flink@10.160.140.216:26073/user/resourcemanager was revoked leadership. Clearing fencing token.
2019-10-09 13:54:21.805 [Curator-ConnectionStateManager-0] INFO  org.apache.flink.runtime.jobmaster.JobManagerRunner  - JobManager for job hdp-shoujiserver_CommercialDotingAccess (2657b25ff073a91c888ff78a28bd2331) was revoked leadership at akka.tcp://flink@10.160.140.216:26073/user/jobmanager_2.
2019-10-09 13:54:21.805 [flink-akka.actor.default-dispatcher-88] INFO  o.a.f.r.leaderretrieval.ZooKeeperLeaderRetrievalService  - Stopping ZooKeeperLeaderRetrievalService /leader/2657b25ff073a91c888ff78a28bd2331/job_manager_lock.
2019-10-09 13:54:21.806 [Curator-ConnectionStateManager-0] WARN  o.a.f.runtime.leaderelection.ZooKeeperLeaderElectionService  - Connection to ZooKeeper suspended. The contender akka.tcp://flink@10.160.140.216:26073/user/jobmanager_2 no longer participates in the leader election.
2019-10-09 13:54:21.806 [flink-akka.actor.default-dispatcher-20] INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher  - Dispatcher akka.tcp://flink@10.160.140.216:26073/user/dispatcher was revoked leadership.
2019-10-09 13:54:21.806 [flink-akka.actor.default-dispatcher-80] INFO  o.a.f.r.leaderretrieval.ZooKeeperLeaderRetrievalService  - Stopping ZooKeeperLeaderRetrievalService /leader/resource_manager_lock.
2019-10-09 13:54:21.806 [flink-akka.actor.default-dispatcher-20] INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher  - Stopping all currently running jobs of dispatcher akka.tcp://flink@10.160.140.216:26073/user/dispatcher.
2019-10-09 13:54:21.806 [Curator-ConnectionStateManager-0] WARN  o.a.f.runtime.leaderelection.ZooKeeperLeaderElectionService  - Connection to ZooKeeper suspended. The contender akka.tcp://flink@10.160.140.216:26073/user/dispatcher no longer participates in the leader election.
2019-10-09 13:54:21.808 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - Job hdp-shoujiserver_CommercialDotingAccess (2657b25ff073a91c888ff78a28bd2331) switched from state RUNNING to SUSPENDING.
org.apache.flink.util.FlinkException: JobManager is no longer the leader.
	at org.apache.flink.runtime.jobmaster.JobManagerRunner.revokeLeadership(JobManagerRunner.java:371)
	at org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService.notLeader(ZooKeeperLeaderElectionService.java:247)
	at org.apache.flink.shaded.curator.org.apache.curator.framework.recipes.leader.LeaderLatch$8.apply(LeaderLatch.java:640)
	at org.apache.flink.shaded.curator.org.apache.curator.framework.recipes.leader.LeaderLatch$8.apply(LeaderLatch.java:636)
	at org.apache.flink.shaded.curator.org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93)
	at org.apache.flink.shaded.curator.org.apache.curator.shaded.com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
	at org.apache.flink.shaded.curator.org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:85)
	at org.apache.flink.shaded.curator.org.apache.curator.framework.recipes.leader.LeaderLatch.setLeadership(LeaderLatch.java:635)
	at org.apache.flink.shaded.curator.org.apache.curator.framework.recipes.leader.LeaderLatch.handleStateChange(LeaderLatch.java:623)
	at org.apache.flink.shaded.curator.org.apache.curator.framework.recipes.leader.LeaderLatch.access$000(LeaderLatch.java:64)
	at org.apache.flink.shaded.curator.org.apache.curator.framework.recipes.leader.LeaderLatch$1.stateChanged(LeaderLatch.java:82)
	at org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager$2.apply(ConnectionStateManager.java:259)
	at org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager$2.apply(ConnectionStateManager.java:255)
	at org.apache.flink.shaded.curator.org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93)
	at org.apache.flink.shaded.curator.org.apache.curator.shaded.com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
	at org.apache.flink.shaded.curator.org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:85)
	at org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:253)
	at org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.flink.shaded.curator.org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:111)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
2019-10-09 13:54:21.806 [flink-akka.actor.default-dispatcher-88] INFO  o.a.flink.runtime.resourcemanager.slotmanager.SlotManager  - Suspending the SlotManager.
2019-10-09 13:54:21.808 [Curator-ConnectionStateManager-0] WARN  o.a.f.r.leaderretrieval.ZooKeeperLeaderRetrievalService  - Connection to ZooKeeper suspended. Can no longer retrieve the leader from ZooKeeper.
2019-10-09 13:54:21.809 [Curator-ConnectionStateManager-0] WARN  o.a.f.runtime.leaderelection.ZooKeeperLeaderElectionService  - Connection to ZooKeeper suspended. The contender akka.tcp://flink@10.160.140.216:26073/user/resourcemanager no longer participates in the leader election.
2019-10-09 13:54:21.819 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - Source: WeiShiDataAnalysisSpout (1/8) (4a825def14a88297d19c9e5738dbf668) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.820 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - Source: WeiShiDataAnalysisSpout (2/8) (6517a467aaf7be56bfb3821b29852e20) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.820 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - Source: WeiShiDataAnalysisSpout (3/8) (7a7ee5b1f3613b97d858564dbfaeb066) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.820 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - Source: WeiShiDataAnalysisSpout (4/8) (2d878a7c86e9d6e0bb0fe737a42037e5) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.821 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - Source: WeiShiDataAnalysisSpout (5/8) (82b545cdf0918a7d14b8c595884a5caf) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.821 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - Source: WeiShiDataAnalysisSpout (6/8) (fb178b7aed87e2fc7c8be11838575cb6) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.823 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - Source: WeiShiDataAnalysisSpout (7/8) (7d2956af9eca83b125a5f2e3d65c7454) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.823 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - Source: WeiShiDataAnalysisSpout (8/8) (8673406c095fbb33ff63b8116c2dc6c4) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.823 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (1/60) (fde9e604f69f27dd1e1afc2d6f96d6a6) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.823 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (2/60) (44429fa0423a6c4e2cfa73b7caf98e7f) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.824 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (3/60) (ccb886b9e7f7f3e9391a66a77a95d936) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.824 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (4/60) (9dfbcca1af51c7565d082ae1d4fbca99) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.824 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (5/60) (984560fc276a70ffb3566ec5236a57d6) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.824 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (6/60) (42c8e83ec209f1947dfc53a3451ca002) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.824 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (7/60) (2fff02384094c4889f8d583db6c7c495) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.825 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (8/60) (bd81e085522ef4915a9413517dcb51ec) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.825 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (9/60) (11762d1b367d402bca99be0624d637c6) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.825 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (10/60) (bcba40c174b64e16b1a48e802e0251c8) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.825 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (11/60) (5655a9923af5f1d40436e2531ca87ff9) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.825 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (12/60) (b9e3017ec604802c151939077f6f4f46) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.825 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (13/60) (4fee14cb460433aa6178fe98315631ff) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.826 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (14/60) (ec80588d10b6a11cc52625f95ebf4858) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.826 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (15/60) (5d55b130964c3c164df9938bb9b9b8e9) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.826 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (16/60) (b67388f779efeb6ffcae57aaaf0e992f) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.826 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (17/60) (c91c36736380adc280b7b9235bd26609) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.826 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (18/60) (f658b30a9c2b4dee63de0fb4797d0c3b) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.827 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (19/60) (bb92a17d4bd6279729ac6e9274ca80ee) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.827 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (20/60) (ecf8a4563990fba29df91ee769e5ea2e) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.827 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (21/60) (8ea7047b457ad4bded12ea4dcea1576b) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.827 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (22/60) (a2cea7a46ecffbd5275d355a29bc09f9) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.827 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (23/60) (105456dc1d11fc9d8164ea0d54794d90) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.828 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (24/60) (faa255028f7bba0ef8b832616a09e0cc) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.828 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (25/60) (80a6454bca3392e1050b02c2b689edae) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.828 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (26/60) (de3561bf14ed9617a4727491ad959219) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.828 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (27/60) (585101b822efc77260c0f0a52eb5f856) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.829 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (28/60) (3205706648aa73f9e34d29c22428ff90) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.829 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (29/60) (24dd134cb75696784bade727d6c57459) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.829 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (30/60) (7d21e0c3934c4bfdc5349f0627fdd923) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.829 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (31/60) (07672677de03585cc630e1014143bec8) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.829 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (32/60) (72159fe698efbaacfb4690b22e4dad03) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.830 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (33/60) (a2f3dce63aac51c82060f02646019af5) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.830 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (34/60) (c5e4f81b8c2590d0f00c7a5c3c05f7a1) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.830 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (35/60) (87579d8547b643a5e1418dca333b426f) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.830 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (36/60) (021f60ef60686370a2e9e6d92fa033ad) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.837 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (37/60) (65b2647fabcbfb1b23aaea63d4627509) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.837 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (38/60) (3f054f2dc5161bd67bbec03d711dea18) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.837 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (39/60) (9b7f93456479e1943de63da8ff8c88d3) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.837 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (40/60) (f2c204e511ed878b8bbd9fded928c215) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.838 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (41/60) (e139c0820ecd8cb8c8fe168b9a847a1e) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.838 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (42/60) (42b961618e0fe859f0761e50e0569113) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.838 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (43/60) (29928821d6f94e715cc2ab444880fb7b) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.838 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (44/60) (83e9b131ba8684474e13236b67916ac3) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.838 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (45/60) (fe3e5b53d7fbc4e795822944317e85bb) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.838 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (46/60) (d7617e20894c359f95cd363faaa3b117) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.839 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (47/60) (e7545083f44b6a8338750542943d67f0) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.839 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (48/60) (0fc95f3ca9dafaeb8d90cd262de82440) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.839 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (49/60) (aa41f35f708625f08bde4b3513e3d769) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.839 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (50/60) (4d01c8d2543d86f2cdd89db1b81df2af) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.839 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (51/60) (6d5314d1d924da1437dfee4ff243ffb9) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.840 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (52/60) (fad4dc6e3ba4d0be2abbecb7a960c914) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.840 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (53/60) (0ccd9b09fc32f61b45031906a7c612b0) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.840 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (54/60) (1fbc38dc9181a399bd30de4e4044e261) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.840 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (55/60) (2ea15174000a056b0d747d1089f6585f) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.841 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (56/60) (28baca2d42222f0a0182c5c89e1717c6) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.841 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (57/60) (e1abb8bce0113fb035f4a36271b73f17) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.841 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (58/60) (9f4a99aad3d396dbbffc75bc9b40f0b6) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.841 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (59/60) (d58fd6d1d88118d0037c3c2e92176b69) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.841 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - ParsingBolt (60/60) (765e9de291878316b4c69000bcd0b2df) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.842 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - DeDuplicationBolt (1/30) (782206df8a2ad0f7a2946eedf0cbc827) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.842 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - DeDuplicationBolt (2/30) (db788aa1e9fdb0022bf9d5a7dc257235) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.842 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - DeDuplicationBolt (3/30) (8567c954df44048c35632a52314d9660) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.842 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - DeDuplicationBolt (4/30) (b8d2a0f72c717e2a06e5770751bf20e7) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.843 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - DeDuplicationBolt (5/30) (d72b93dbfa363c38bd6b7451a39c7d67) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.843 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - DeDuplicationBolt (6/30) (559d7f2c0bcb8255df8963b3b4329f66) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.843 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - DeDuplicationBolt (7/30) (8124a163b7727ce53601ba21c7a49acc) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.843 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - DeDuplicationBolt (8/30) (c151f0d09157158e3f449a847c41cc65) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.843 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - DeDuplicationBolt (9/30) (86b2c2178b48d7bb74742036b6489260) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.843 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - DeDuplicationBolt (10/30) (f716beb167fbe51c1db7798c1f1794d3) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.844 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - DeDuplicationBolt (11/30) (b72e2b66c4c3fa24a2e6d247210f9e34) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.844 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - DeDuplicationBolt (12/30) (e63a7ed9a8fb828a1cda0e206fa803e1) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.844 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - DeDuplicationBolt (13/30) (7dcb47fd7e88bdc86ae45344d9779e99) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.844 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - DeDuplicationBolt (14/30) (ccc164979332f39f374664d758345301) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.844 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - DeDuplicationBolt (15/30) (b21287c5c6b9b89124a1cd6810eb2a21) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.844 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - DeDuplicationBolt (16/30) (91071077d5e04266d5b6d9833cf2f9e4) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.845 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - DeDuplicationBolt (17/30) (b5ec2ff0c6decca89ff76223c25b9d9b) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.845 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - DeDuplicationBolt (18/30) (0774604957ec0d2680b8673b98dbbe7b) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.845 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - DeDuplicationBolt (19/30) (6e402ae9f4d0b8755c0c7a2ef67f2c1a) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.845 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - DeDuplicationBolt (20/30) (51493f15a4b7166aa390f854a597c4f6) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.845 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - DeDuplicationBolt (21/30) (e329d85d95a8a6a142b8157bd3b2eeee) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.845 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - DeDuplicationBolt (22/30) (c17a752c676bd4fa1559b6641f959fc7) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.846 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - DeDuplicationBolt (23/30) (94bfdd3b1bb3be163c951b41aa80f5a8) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.846 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - DeDuplicationBolt (24/30) (f4a378189dd403de5d4d5a3e77415f9e) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.846 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - DeDuplicationBolt (25/30) (d16ff3ac286ab1fd32caa9f46fca5619) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.846 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - DeDuplicationBolt (26/30) (090c3aaa6bb23261e74a7bf70060a36d) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.847 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - DeDuplicationBolt (27/30) (9aeebb1f3b6d51be5388674955953c52) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.847 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - DeDuplicationBolt (28/30) (23c2fceab79f540a9d89e2e4f092a865) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.847 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - DeDuplicationBolt (29/30) (e6bb884157821902ae36d036737efaa5) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.847 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - DeDuplicationBolt (30/30) (afca1c29fa6c50a49c88d59d3df459f6) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.847 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - StatisticsBolt (1/30) (9adb75badd9cb8afbb4251a33ef27171) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.848 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - StatisticsBolt (2/30) (473835002699bfb37c5b88edafd219a1) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.848 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - StatisticsBolt (3/30) (e7e030cb12de97e633cf473d7e627206) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.848 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - StatisticsBolt (4/30) (0459e2f2ea079437f62769f2aee861d2) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.848 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - StatisticsBolt (5/30) (ed428b67cd250fd6407a0b5243aab514) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.848 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - StatisticsBolt (6/30) (4a54f08765d6c9faac31ace85a406819) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.849 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - StatisticsBolt (7/30) (24ab50af8fdab9de5f0915479cedc50f) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.849 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - StatisticsBolt (8/30) (5f365913e8e69f0bb8fad4fcac0337ea) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.849 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - StatisticsBolt (9/30) (d4ac35d28253d1dd096fa6bab4fafc8a) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.849 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - StatisticsBolt (10/30) (1ad00f44a299183f81f61ed829410b1d) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.849 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - StatisticsBolt (11/30) (98238896820682b892815dfa2f62e6c2) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.849 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - StatisticsBolt (12/30) (6019139a62f395abb0d3bb0fa51b326e) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.850 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - StatisticsBolt (13/30) (73f72dd701d042dacd95988521873013) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.850 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - StatisticsBolt (14/30) (3109dae5918d772b64e7d8b28b128dfe) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.850 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - StatisticsBolt (15/30) (10c0effec4da40afe144a59556882680) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.850 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - StatisticsBolt (16/30) (722f04d080fed0df95787fc11a7c95d0) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.850 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - StatisticsBolt (17/30) (07461d9203eedf28c0a8c893a631ca2f) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.850 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - StatisticsBolt (18/30) (206c625688c12fa82cfc3bae7b60f24d) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.851 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - StatisticsBolt (19/30) (6fd477a3d452423a09cd0ba0494a6deb) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.851 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - StatisticsBolt (20/30) (d0aa4c46dbb46db09c9395e39b41a4c1) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.851 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - StatisticsBolt (21/30) (0dde7060b76feb5e7333e20d3fe4e7a1) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.851 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - StatisticsBolt (22/30) (ce6749c3e5b996df8647603e4e8fbfdb) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.851 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - StatisticsBolt (23/30) (8270b98412ef6613859f9ae0109850e3) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.851 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - StatisticsBolt (24/30) (136806c81715dd66a3d5cf8f49ceb031) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.852 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - StatisticsBolt (25/30) (e95b22dff10204581a86a35461bf5768) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.852 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - StatisticsBolt (26/30) (9fff9a9c688f9e383f18661923a814ba) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.852 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - StatisticsBolt (27/30) (2626c8dad0fbab32fadd995930211d8a) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.852 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - StatisticsBolt (28/30) (70aa1e618f77758c684cae15b7722752) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.852 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - StatisticsBolt (29/30) (ef3392ed8088af2c05212e0fb0a5aac6) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.853 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - StatisticsBolt (30/30) (6af3938acd0f635d95c9c432edeab539) switched from RUNNING to CANCELING.
2019-10-09 13:54:21.855 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.jobmaster.JobMaster  - Close ResourceManager connection 490c7cf7fc01baf7e0ece283d48690a4: JobManager is no longer the leader..
2019-10-09 13:54:21.855 [flink-akka.actor.default-dispatcher-88] INFO  org.apache.flink.runtime.jobmaster.slotpool.SlotPool  - Suspending SlotPool.
2019-10-09 13:54:21.856 [flink-akka.actor.default-dispatcher-80] INFO  org.apache.flink.runtime.jobmaster.JobMaster  - Stopping the JobMaster for job hdp-shoujiserver_CommercialDotingAccess(2657b25ff073a91c888ff78a28bd2331).
2019-10-09 13:54:21.856 [flink-akka.actor.default-dispatcher-20] INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcActor  - The rpc endpoint org.apache.flink.runtime.jobmaster.slotpool.SlotPool has not been started yet. Discarding message org.apache.flink.runtime.rpc.messages.LocalRpcInvocation until processing is started.
2019-10-09 13:54:21.856 [flink-akka.actor.default-dispatcher-20] INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcActor  - The rpc endpoint org.apache.flink.runtime.jobmaster.slotpool.SlotPool has not been started yet. Discarding message org.apache.flink.runtime.rpc.messages.LocalRpcInvocation until processing is started.
2019-10-09 13:54:21.857 [flink-akka.actor.default-dispatcher-20] INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcActor  - The rpc endpoint org.apache.flink.runtime.jobmaster.slotpool.SlotPool has not been started yet. Discarding message org.apache.flink.runtime.rpc.messages.LocalRpcInvocation until processing is started.
2019-10-09 13:54:21.857 [flink-akka.actor.default-dispatcher-20] INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcActor  - The rpc endpoint org.apache.flink.runtime.jobmaster.slotpool.SlotPool has not been started yet. Discarding message org.apache.flink.runtime.rpc.messages.LocalRpcInvocation until processing is started.
2019-10-09 13:54:21.857 [flink-akka.actor.default-dispatcher-20] INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcActor  - The rpc endpoint org.apache.flink.runtime.jobmaster.slotpool.SlotPool has not been started yet. Discarding message org.apache.flink.runtime.rpc.messages.LocalRpcInvocation until processing is started.
2019-10-09 13:54:21.857 [flink-akka.actor.default-dispatcher-20] INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcActor  - The rpc endpoint org.apache.flink.runtime.jobmaster.slotpool.SlotPool has not been started yet. Discarding message org.apache.flink.runtime.rpc.messages.LocalRpcInvocation until processing is started.
2019-10-09 13:54:21.857 [flink-akka.actor.default-dispatcher-20] INFO  org.apache.flink.runtime.jobmaster.slotpool.SlotPool  - Stopping SlotPool.
2019-10-09 13:54:21.857 [flink-akka.actor.default-dispatcher-20] INFO  o.a.f.runtime.leaderelection.ZooKeeperLeaderElectionService  - Stopping ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/2657b25ff073a91c888ff78a28bd2331/job_manager_lock'}.
2019-10-09 13:54:22.602 [main-SendThread(10.160.101.29:2181)] WARN  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/tmp/jaas-4748011049181233444.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2019-10-09 13:54:22.603 [main-SendThread(10.160.101.29:2181)] INFO  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Opening socket connection to server 10.160.101.29/10.160.101.29:2181
2019-10-09 13:54:22.603 [main-EventThread] ERROR o.a.flink.shaded.curator.org.apache.curator.ConnectionState  - Authentication failed
2019-10-09 13:54:22.603 [main-SendThread(10.160.101.29:2181)] WARN  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Session 0x4597758a3589ca6 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: 拒绝连接
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
	at org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
2019-10-09 13:54:22.921 [main-SendThread(10.160.101.28:2181)] WARN  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/tmp/jaas-4748011049181233444.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2019-10-09 13:54:22.922 [main-SendThread(10.160.101.28:2181)] INFO  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Opening socket connection to server 10.160.101.28/10.160.101.28:2181
2019-10-09 13:54:22.922 [main-EventThread] ERROR o.a.flink.shaded.curator.org.apache.curator.ConnectionState  - Authentication failed
2019-10-09 13:54:22.922 [main-SendThread(10.160.101.28:2181)] WARN  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Session 0x4597758a3589ca6 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: 拒绝连接
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
	at org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
2019-10-09 13:54:24.008 [main-SendThread(10.160.101.26:2181)] WARN  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/tmp/jaas-4748011049181233444.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2019-10-09 13:54:24.008 [main-SendThread(10.160.101.26:2181)] INFO  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Opening socket connection to server 10.160.101.26/10.160.101.26:2181
2019-10-09 13:54:24.008 [main-EventThread] ERROR o.a.flink.shaded.curator.org.apache.curator.ConnectionState  - Authentication failed
2019-10-09 13:54:24.009 [main-SendThread(10.160.101.26:2181)] WARN  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Session 0x4597758a3589ca6 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: 拒绝连接
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
	at org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
2019-10-09 13:54:24.778 [main-SendThread(10.160.101.30:2181)] WARN  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/tmp/jaas-4748011049181233444.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2019-10-09 13:54:24.778 [main-SendThread(10.160.101.30:2181)] INFO  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Opening socket connection to server 10.160.101.30/10.160.101.30:2181
2019-10-09 13:54:24.778 [main-EventThread] ERROR o.a.flink.shaded.curator.org.apache.curator.ConnectionState  - Authentication failed
2019-10-09 13:54:24.779 [main-SendThread(10.160.101.30:2181)] WARN  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Session 0x4597758a3589ca6 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: 拒绝连接
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
	at org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
2019-10-09 13:54:26.748 [main-SendThread(10.160.101.27:2181)] WARN  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/tmp/jaas-4748011049181233444.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2019-10-09 13:54:26.748 [main-SendThread(10.160.101.27:2181)] INFO  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Opening socket connection to server 10.160.101.27/10.160.101.27:2181
2019-10-09 13:54:26.749 [main-EventThread] ERROR o.a.flink.shaded.curator.org.apache.curator.ConnectionState  - Authentication failed
2019-10-09 13:54:26.749 [main-SendThread(10.160.101.27:2181)] INFO  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Socket connection established to 10.160.101.27/10.160.101.27:2181, initiating session
2019-10-09 13:54:26.751 [main-SendThread(10.160.101.27:2181)] INFO  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Unable to read additional data from server sessionid 0x4597758a3589ca6, likely server has closed socket, closing socket connection and attempting reconnect
2019-10-09 13:54:27.673 [main-SendThread(10.160.101.29:2181)] WARN  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/tmp/jaas-4748011049181233444.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2019-10-09 13:54:27.674 [main-SendThread(10.160.101.29:2181)] INFO  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Opening socket connection to server 10.160.101.29/10.160.101.29:2181
2019-10-09 13:54:27.674 [main-EventThread] ERROR o.a.flink.shaded.curator.org.apache.curator.ConnectionState  - Authentication failed
2019-10-09 13:54:27.674 [main-SendThread(10.160.101.29:2181)] INFO  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Socket connection established to 10.160.101.29/10.160.101.29:2181, initiating session
2019-10-09 13:54:27.675 [main-SendThread(10.160.101.29:2181)] INFO  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Unable to read additional data from server sessionid 0x4597758a3589ca6, likely server has closed socket, closing socket connection and attempting reconnect
2019-10-09 13:54:27.865 [main-SendThread(10.160.101.28:2181)] WARN  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/tmp/jaas-4748011049181233444.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2019-10-09 13:54:27.865 [main-SendThread(10.160.101.28:2181)] INFO  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Opening socket connection to server 10.160.101.28/10.160.101.28:2181
2019-10-09 13:54:27.865 [main-EventThread] ERROR o.a.flink.shaded.curator.org.apache.curator.ConnectionState  - Authentication failed
2019-10-09 13:54:27.866 [main-SendThread(10.160.101.28:2181)] INFO  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Socket connection established to 10.160.101.28/10.160.101.28:2181, initiating session
2019-10-09 13:54:27.867 [main-SendThread(10.160.101.28:2181)] INFO  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Unable to read additional data from server sessionid 0x4597758a3589ca6, likely server has closed socket, closing socket connection and attempting reconnect
2019-10-09 13:54:28.078 [main-SendThread(10.160.101.26:2181)] WARN  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/tmp/jaas-4748011049181233444.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2019-10-09 13:54:28.078 [main-SendThread(10.160.101.26:2181)] INFO  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Opening socket connection to server 10.160.101.26/10.160.101.26:2181
2019-10-09 13:54:28.079 [main-EventThread] ERROR o.a.flink.shaded.curator.org.apache.curator.ConnectionState  - Authentication failed
2019-10-09 13:54:28.079 [main-SendThread(10.160.101.26:2181)] INFO  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Socket connection established to 10.160.101.26/10.160.101.26:2181, initiating session
2019-10-09 13:54:28.080 [main-SendThread(10.160.101.26:2181)] INFO  o.a.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn  - Session establishment complete on server 10.160.101.26/10.160.101.26:2181, sessionid = 0x4597758a3589ca6, negotiated timeout = 40000
2019-10-09 13:54:28.081 [main-EventThread] INFO  o.a.f.s.c.o.a.curator.framework.state.ConnectionStateManager  - State change: RECONNECTED
2019-10-09 13:54:28.081 [Curator-ConnectionStateManager-0] INFO  o.a.f.runtime.leaderelection.ZooKeeperLeaderElectionService  - Connection to ZooKeeper was reconnected. Leader election can be restarted.
2019-10-09 13:54:28.082 [Curator-PathChildrenCache-0] INFO  o.a.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore  - ZooKeeper connection RECONNECTED. Changes to the submitted job graphs are monitored again.
2019-10-09 13:54:28.082 [Curator-ConnectionStateManager-0] INFO  o.a.f.r.leaderretrieval.ZooKeeperLeaderRetrievalService  - Connection to ZooKeeper was reconnected. Leader retrieval can be restarted.
2019-10-09 13:54:28.082 [Curator-ConnectionStateManager-0] INFO  o.a.f.runtime.leaderelection.ZooKeeperLeaderElectionService  - Connection to ZooKeeper was reconnected. Leader election can be restarted.
2019-10-09 13:54:28.083 [Curator-ConnectionStateManager-0] INFO  o.a.f.r.leaderretrieval.ZooKeeperLeaderRetrievalService  - Connection to ZooKeeper was reconnected. Leader retrieval can be restarted.
2019-10-09 13:54:28.083 [Curator-ConnectionStateManager-0] INFO  o.a.f.runtime.leaderelection.ZooKeeperLeaderElectionService  - Connection to ZooKeeper was reconnected. Leader election can be restarted.
2019-10-09 13:54:28.087 [flink-akka.actor.default-dispatcher-88] INFO  o.a.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore  - Released locks of job graph 2657b25ff073a91c888ff78a28bd2331 from ZooKeeper.
2019-10-09 13:54:28.088 [main-EventThread] INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint  - http://10.160.140.216:52303 was granted leadership with leaderSessionID=d80d351c-56ca-490f-aa78-681847d58cfe
2019-10-09 13:54:28.092 [flink-akka.actor.default-dispatcher-83] INFO  org.apache.flink.yarn.YarnResourceManager  - ResourceManager akka.tcp://flink@10.160.140.216:26073/user/resourcemanager was granted leadership with fencing token 8b328120280016029994e898e40749f8
2019-10-09 13:54:28.092 [flink-akka.actor.default-dispatcher-83] INFO  o.a.flink.runtime.resourcemanager.slotmanager.SlotManager  - Starting the SlotManager.
2019-10-09 13:54:28.092 [flink-akka.actor.default-dispatcher-48] INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher  - Dispatcher akka.tcp://flink@10.160.140.216:26073/user/dispatcher was granted leadership with fencing token 65d868d3-0665-4555-ae75-86975bb3c7a9
2019-10-09 13:54:28.093 [flink-akka.actor.default-dispatcher-72] INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher  - Recovering all persisted jobs.
2019-10-09 13:54:28.130 [flink-akka.actor.default-dispatcher-72] INFO  o.a.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore  - Recovered SubmittedJobGraph(2657b25ff073a91c888ff78a28bd2331, null).
2019-10-09 13:54:28.212 [flink-akka.actor.default-dispatcher-72] INFO  org.apache.flink.yarn.YarnResourceManager  - Registering TaskManager with ResourceID container_e64_1564717237973_1572220_01_000003 (akka.tcp://flink@10.160.167.236:37058/user/taskmanager_0) at ResourceManager
2019-10-09 13:54:28.515 [flink-akka.actor.default-dispatcher-72] INFO  org.apache.flink.yarn.YarnResourceManager  - Registering TaskManager with ResourceID container_e64_1564717237973_1572220_01_000005 (akka.tcp://flink@10.162.92.119:34318/user/taskmanager_0) at ResourceManager
2019-10-09 13:54:29.072 [flink-akka.actor.default-dispatcher-72] INFO  org.apache.flink.yarn.YarnResourceManager  - Registering TaskManager with ResourceID container_e64_1564717237973_1572220_01_000008 (akka.tcp://flink@10.160.132.173:48920/user/taskmanager_0) at ResourceManager
2019-10-09 13:54:29.473 [flink-akka.actor.default-dispatcher-88] INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService  - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/jobmanager_3 .
2019-10-09 13:54:29.474 [flink-akka.actor.default-dispatcher-88] INFO  org.apache.flink.runtime.jobmaster.JobMaster  - Initializing job hdp-shoujiserver_CommercialDotingAccess (2657b25ff073a91c888ff78a28bd2331).
2019-10-09 13:54:29.487 [flink-akka.actor.default-dispatcher-88] INFO  org.apache.flink.runtime.jobmaster.JobMaster  - Using restart strategy FailureRateRestartStrategy(failuresInterval=3600000 msdelayInterval=30000 msmaxFailuresPerInterval=3) for hdp-shoujiserver_CommercialDotingAccess (2657b25ff073a91c888ff78a28bd2331).
2019-10-09 13:54:29.489 [flink-akka.actor.default-dispatcher-88] INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService  - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.slotpool.SlotPool at akka://flink/user/25b2e77f-fe0c-4e2d-af38-8900a36077d7 .
2019-10-09 13:54:29.489 [flink-akka.actor.default-dispatcher-88] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - Job recovers via failover strategy: full graph restart
2019-10-09 13:54:29.490 [flink-akka.actor.default-dispatcher-88] INFO  org.apache.flink.runtime.jobmaster.JobMaster  - Running initialization on master for job hdp-shoujiserver_CommercialDotingAccess (2657b25ff073a91c888ff78a28bd2331).
2019-10-09 13:54:29.490 [flink-akka.actor.default-dispatcher-88] INFO  org.apache.flink.runtime.jobmaster.JobMaster  - Successfully ran initialization on master in 0 ms.
2019-10-09 13:54:29.497 [flink-akka.actor.default-dispatcher-88] INFO  o.a.f.runtime.checkpoint.ZooKeeperCompletedCheckpointStore  - Initialized in '/checkpoints/2657b25ff073a91c888ff78a28bd2331'.
2019-10-09 13:54:29.498 [flink-akka.actor.default-dispatcher-88] INFO  org.apache.flink.runtime.jobmaster.JobMaster  - Loading state backend via factory org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory
2019-10-09 13:54:29.498 [flink-akka.actor.default-dispatcher-88] WARN  org.apache.flink.configuration.Configuration  - Config uses deprecated configuration key 'state.backend.rocksdb.checkpointdir' instead of proper key 'state.backend.rocksdb.localdir'
2019-10-09 13:54:29.502 [flink-akka.actor.default-dispatcher-88] INFO  o.a.f.runtime.checkpoint.ZooKeeperCompletedCheckpointStore  - Recovering checkpoints from ZooKeeper.
2019-10-09 13:54:29.513 [flink-akka.actor.default-dispatcher-88] INFO  o.a.f.runtime.checkpoint.ZooKeeperCompletedCheckpointStore  - Found 5 checkpoints in ZooKeeper.
2019-10-09 13:54:29.514 [flink-akka.actor.default-dispatcher-88] INFO  o.a.f.runtime.checkpoint.ZooKeeperCompletedCheckpointStore  - Trying to fetch 5 checkpoints from storage.
2019-10-09 13:54:29.514 [flink-akka.actor.default-dispatcher-88] INFO  o.a.f.runtime.checkpoint.ZooKeeperCompletedCheckpointStore  - Trying to retrieve checkpoint 33036.
2019-10-09 13:54:29.516 [flink-akka.actor.default-dispatcher-21] INFO  org.apache.flink.yarn.YarnResourceManager  - Registering TaskManager with ResourceID container_e64_1564717237973_1572220_01_000010 (akka.tcp://flink@10.160.124.173:30267/user/taskmanager_0) at ResourceManager
2019-10-09 13:54:29.523 [flink-akka.actor.default-dispatcher-88] INFO  o.a.f.runtime.checkpoint.ZooKeeperCompletedCheckpointStore  - Trying to retrieve checkpoint 33037.
2019-10-09 13:54:29.532 [flink-akka.actor.default-dispatcher-88] INFO  o.a.f.runtime.checkpoint.ZooKeeperCompletedCheckpointStore  - Trying to retrieve checkpoint 33038.
2019-10-09 13:54:29.540 [flink-akka.actor.default-dispatcher-88] INFO  o.a.f.runtime.checkpoint.ZooKeeperCompletedCheckpointStore  - Trying to retrieve checkpoint 33039.
2019-10-09 13:54:29.549 [flink-akka.actor.default-dispatcher-88] INFO  o.a.f.runtime.checkpoint.ZooKeeperCompletedCheckpointStore  - Trying to retrieve checkpoint 33040.
2019-10-09 13:54:30.712 [flink-akka.actor.default-dispatcher-88] INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator  - Restoring job 2657b25ff073a91c888ff78a28bd2331 from latest valid checkpoint: Checkpoint 33040 @ 1570600391503 for 2657b25ff073a91c888ff78a28bd2331.
2019-10-09 13:54:30.715 [flink-akka.actor.default-dispatcher-88] INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator  - No master state to restore
2019-10-09 13:54:30.715 [flink-akka.actor.default-dispatcher-88] INFO  o.a.f.runtime.leaderelection.ZooKeeperLeaderElectionService  - Starting ZooKeeperLeaderElectionService ZooKeeperLeaderElectionService{leaderPath='/leader/2657b25ff073a91c888ff78a28bd2331/job_manager_lock'}.
2019-10-09 13:54:30.721 [main-EventThread] INFO  org.apache.flink.runtime.jobmaster.JobManagerRunner  - JobManager runner for job hdp-shoujiserver_CommercialDotingAccess (2657b25ff073a91c888ff78a28bd2331) was granted leadership with session id 829a1a30-be5a-4e4a-aa0c-21cc71b45599 at akka.tcp://flink@10.160.140.216:26073/user/jobmanager_3.
2019-10-09 13:54:30.724 [flink-akka.actor.default-dispatcher-21] INFO  o.a.f.r.leaderretrieval.ZooKeeperLeaderRetrievalService  - Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock.
2019-10-09 13:54:30.724 [flink-akka.actor.default-dispatcher-21] INFO  org.apache.flink.runtime.jobmaster.JobMaster  - Starting execution of job hdp-shoujiserver_CommercialDotingAccess (2657b25ff073a91c888ff78a28bd2331)
2019-10-09 13:54:30.725 [flink-akka.actor.default-dispatcher-21] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - Job hdp-shoujiserver_CommercialDotingAccess (2657b25ff073a91c888ff78a28bd2331) switched from state CREATED to RUNNING.
2019-10-09 13:54:30.725 [flink-akka.actor.default-dispatcher-21] INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph  - Source: WeiShiDataAnalysisSpout (1/8) (d072a1bc3f3dbf7896ef619951d91c88) switched from CREATED to SCHEDULED.
After the zk reconnected after a suspend, the job was recovered, and you can see, there was a StandaloneDispatcher created.


Don't know why the 'FencingTokenException: Fencing token not set' exception happened, and don't know why different dispatchers were created.

Thanks

Joshua


On Wed, Oct 2, 2019 at 9:10 PM Till Rohrmann <trohrmann@apache.org> wrote:
Hi Bruce, are you able to provide us with the full debug logs? From the excerpt itself it is hard to tell what is going on.

Cheers,
Till

On Wed, Oct 2, 2019 at 2:24 PM Fabian Hueske <fhueske@gmail.com> wrote:
Hi Bruce,

I haven't seen such an exception yet, but maybe Till (in CC) can help.

Best,
Fabian

Am Di., 1. Okt. 2019 um 05:51 Uhr schrieb Hanson, Bruce <bruce.hanson@here.com>:

Hi all,

 

We are running some of our Flink jobs with Job Manager High Availability. Occasionally we get a cluster that comes up improperly and doesn’t respond. Attempts to submit the job seem to hang and when we hit the /overview REST endpoint in the Job Manager we get a 500 error and a fencing token exception like this:

 

2019-09-21 05:04:07.785 [flink-akka.actor.default-dispatcher-4428] level=ERROR o.a.f.runtime.rest.handler.cluster.ClusterOverviewHandler  - Implementation error: Unhandled exception.

org.apache.flink.runtime.rpc.exceptions.FencingTokenException: Fencing token not set: Ignoring message LocalFencedMessage(null, LocalRpcInvocation(requestResourceOverview(Time))) sent to akka.tcp://flink@job-ef80a156-3350-4e85-8761-b0e42edc346f-jm-0.job-ef80a156-3350-4e85-8761-b0e42edc346f-jm-svc.olp-here-test-j-ef80a156-3350-4e85-8761-b0e42edc346f.svc.cluster.local:6126/user/resourcemanager because the fencing token is null.

        at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:59)

        at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:142)

        at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.onReceive(FencedAkkaRpcActor.java:40)

        at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)

        at akka.actor.Actor$class.aroundReceive(Actor.scala:502)

        at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)

        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)

        at akka.actor.ActorCell.invoke(ActorCell.scala:495)

        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)

        at akka.dispatch.Mailbox.run(Mailbox.scala:224)

        at akka.dispatch.Mailbox.exec(Mailbox.scala:234)

        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)

        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)

        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)

        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

 

 

We are running Flink 1.7.1 in Kubernetes and run each job in its own namespace with a three-node Zookeeper cluster and two Job Managers, plus one or more Task Managers. I have been able to replicate the issue, but don’t find any difference in the logs between a failing cluster and a good one.

 

Does anyone here have any ideas as to what’s happening, or what I should be looking for?

 

-Bruce

 

 

 

 

cid:image001.png@01D2B473.0F7F85E0

 

Bruce Hanson

Principal Engineer

M: +1 425 681 0422

 

HERE Technologies

701 Pike Street, Suite 2000

Seattle, WA 98101 USA

47° 36' 41" N 122° 19' 57" W

 

cid:image002.png@01D2B473.0F7F85E0    cid:image003.png@01D2B473.0F7F85E0   cid:image004.png@01D2B473.0F7F85E0    cid:image005.png@01D2B473.0F7F85E0    cid:image006.png@01D2B473.0F7F85E0