flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nico Kruber <n...@data-artisans.com>
Subject Re: Can't get my job restarted on job manager failures
Date Tue, 20 Jun 2017 16:24:59 GMT
yes, this supports my theory: both the ResourceManager and the TaskManager 
instances are trying to reconnect to the old JobManager to re-establish a link 
in case of temporary failures.
* In the best case, the connection is re-established and everything continues.
* Since your failure is permanent, this is not the case and after some 
retries, Flink also thinks that ;)
We can't really avoid the warnings though, even in a HA setup, since the 
administrator might want to know what is happening and what Flink is doing 
under the hood.

Everything's fine then.

Nico

On Tuesday, 20 June 2017 17:56:00 CEST Mikhail Pryakhin wrote:
> Thanks a lot Niko!
> 
> It is definitely a permanent failure, I’ve tried multiple times and always
> got the same behaviour. And as you said, right after the job manager has
> started the errors stop as well.
> 
> I also checked the logs of the Task Manager and found huge amount of the
> following warnings ():
> 
> 2017-06-20 18:38:45,087 WARN  akka.remote.ReliableDeliverySupervisor        
>                - Association with remote system
> [akka.tcp://flink@dmpkit-dev-dn2:30086] has failed, address is now gated
> for [5000] ms. Reason: [Association failed with
> [akka.tcp://flink@dmpkit-dev-dn2:30086]] Caused by: [Connection refused:
> dmpkit-dev-dn2/136.243.170.205:30086] 2017-06-20 18:38:50,105 WARN 
> akka.remote.ReliableDeliverySupervisor                        - Association
> with remote system [akka.tcp://flink@dmpkit-dev-dn2:30086] has failed,
> address is now gated for [5000] ms. Reason: [Association failed with
> [akka.tcp://flink@dmpkit-dev-dn2:30086]] Caused by: [Connection refused:
> dmpkit-dev-dn2/136.243.170.205:30086] 2017-06-20 18:38:55,125 WARN 
> akka.remote.ReliableDeliverySupervisor                        - Association
> with remote system [akka.tcp://flink@dmpkit-dev-dn2:30086] has failed,
> address is now gated for [5000] ms. Reason: [Association failed with
> [akka.tcp://flink@dmpkit-dev-dn2:30086]] Caused by: [Connection refused:
> dmpkit-dev-dn2/136.243.170.205:30086]
> 
> the warnings above finally finish with the error message
> 
> 2017-06-20 18:41:30,744 ERROR Remoting                                      
>                - Association to [akka.tcp://flink@dmpkit-dev-dn2:30086]
> with UID [77777149] irrecoverably failed. Quarantining address.
> java.util.concurrent.TimeoutException: Delivery of system messages timed
> out and they were dropped. at
> akka.remote.ReliableDeliverySupervisor$$anonfun$gated$1.applyOrElse(Endpoin
> t.scala:336) at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
> 	at akka.remote.ReliableDeliverySupervisor.aroundReceive(Endpoint.scala:
189)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
> 	at akka.actor.ActorCell.invoke(ActorCell.scala:487)
> 	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
> 	at akka.dispatch.Mailbox.run(Mailbox.scala:220)
> 	at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDi
> spatcher.scala:397) at
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:
> 1339) at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.jav
> a:107)
> 
> After that multiple Info lines follow.
> 
> 2017-06-20 18:47:40,453 INFO 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator  - Marking
> the coordinator 2147483480 dead.
> 
> 
> 
> 
> 
> Kind Regards,
> Mike Pryakhin
> 
> > On 20 Jun 2017, at 18:27, Nico Kruber <nico@data-artisans.com> wrote:
> > 
> > My best guess here is that the resource manager is still trying to connect
> > to the JobManager which failed - after all, how should it know whether
> > this is a temporary network failure or a permanent failure?!
> > 
> > If, after your new JobManager starts, the errors stop as well, I'd say,
> > you
> > don't have to worry about the messages.
> > Till (cc'd) may elaborate a bit more on this.
> > 
> > 
> > Nico
> > 
> > On Tuesday, 20 June 2017 17:06:00 CEST Mikhail Pryakhin wrote:
> >> Hi Niko,
> >> Thanks for your reply!
> >> 
> >> Having zookeeper-related properties configured everything works smoothly!
> >> I was confused because the doc references high availability configuration
> >> for yarn session mode, anyway thanks a lot!
> >> 
> >> Now I noticed another problem, when I kill the job manager, it then gets
> >> restarted by YARN, and the following stacktrace appears in the Job
> >> Manager
> >> log:
> >> 
> >> 2017-06-20 16:44:53,843 WARN
> >> org.apache.flink.runtime.webmonitor.JobManagerRetriever       - Failed to
> >> retrieve leader gateway and port. akka.actor.ActorNotFound: Actor not
> >> found
> >> for: ActorSelection[Anchor(akka.tcp://flink@dmpkit-dev-dn2:24660/),
> >> Path(/user/jobmanager)] at
> >> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scal
> >> a:
> >> 65) at
> >> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scal
> >> a:
> >> 63) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
> >> 
> >> 	at
> >> 
> >> akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecuto
> >> r.
> >> scala:55) at
> >> akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:73) at
> >> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExec
> >> ut
> >> e(Future.scala:74) at
> >> akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:120)
> >> at
> >> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Futur
> >> e.
> >> scala:73) at
> >> scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
> >> at
> >> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:24
> >> 8)
> >> at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:266)
> >> 
> >> 	at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:533)
> >> 	at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:569)
> >> 	at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:559)
> >> 	at
> >> 
> >> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteA
> >> ct
> >> orRefProvider.scala:87) at
> >> akka.remote.EndpointWriter.postStop(Endpoint.scala:583)
> >> 
> >> 	at akka.actor.Actor$class.aroundPostStop(Actor.scala:477)
> >> 	at akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:437)
> >> 	at
> >> 
> >> akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$
> >> fi
> >> nishTerminate(FaultHandling.scala:210) at
> >> akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172)
> >> at akka.actor.ActorCell.terminate(ActorCell.scala:369)
> >> 
> >> 	at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462)
> >> 	at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
> >> 	at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
> >> 	at akka.dispatch.Mailbox.run(Mailbox.scala:219)
> >> 	at
> >> 
> >> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(Abstract
> >> Di
> >> spatcher.scala:397) at
> >> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at
> >> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.jav
> >> a:
> >> 1339) at
> >> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> >> at
> >> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.j
> >> av
> >> a:107) 2017-06-20 16:44:53,932 INFO
> >> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl         - Received
> >> new token for : dmpkit-dev-dn2:8041 2017-06-20 16:44:53,937 INFO
> >> org.apache.flink.yarn.YarnFlinkResourceManager                - Retrieved
> >> 1
> >> TaskManagers from previous attempt 2017-06-20 16:44:53,948 INFO
> >> org.apache.flink.yarn.YarnFlinkResourceManager                - Trying to
> >> associate with JobManager leader
> >> akka.tcp://flink@dmpkit-dev-dn2:24660/user/jobmanager 2017-06-20
> >> 16:45:03,970 ERROR org.apache.flink.yarn.YarnFlinkResourceManager
> >> 
> >>     - Resource manager could not register at JobManager
> >> 
> >> akka.pattern.AskTimeoutException: Ask timed out on
> >> [ActorSelection[Anchor(akka.tcp://flink@dmpkit-dev-dn2:24660/),
> >> Path(/user/jobmanager)]] after [10000 ms] at
> >> akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:334
> >> )
> >> at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117)
> >> 
> >> 	at
> >> 
> >> scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future
> >> .s
> >> cala:599) at
> >> scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:10
> >> 9)
> >> at
> >> scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:59
> >> 7)
> >> at
> >> akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.s
> >> ca
> >> la:474) at
> >> akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.
> >> sc
> >> ala:425) at
> >> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:4
> >> 29
> >> ) at
> >> akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:381)
> >> at java.lang.Thread.run(Thread.java:745)
> >> 2017-06-20 16:45:03,971 INFO 
> >> org.apache.flink.yarn.YarnFlinkResourceManager>> 
> >>               - Trying to associate with JobManager leader
> >> 
> >> akka.tcp://flink@dmpkit-dev-dn2:24660/user/jobmanager 2017-06-20
> >> 16:45:03,975 WARN  akka.remote.ReliableDeliverySupervisor
> >> 
> >>     - Association with remote system
> >> 
> >> [akka.tcp://flink@dmpkit-dev-dn2:24660] has failed, address is now gated
> >> for [5000] ms. Reason: [Association failed with
> >> [akka.tcp://flink@dmpkit-dev-dn2:24660]] Caused by: [Connection refused:
> >> dmpkit-dev-dn2/136.243.170.205:24660] 2017-06-20 16:45:13,989 ERROR
> >> org.apache.flink.yarn.YarnFlinkResourceManager                - Resource
> >> manager could not register at JobManager
> >> akka.pattern.AskTimeoutException:
> >> Ask timed out on
> >> [ActorSelection[Anchor(akka.tcp://flink@dmpkit-dev-dn2:24660/),
> >> Path(/user/jobmanager)]] after [10000 ms] at
> >> akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:334
> >> )
> >> at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117)
> >> 
> >> 	at
> >> 
> >> scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future
> >> .s
> >> cala:599) at
> >> scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:10
> >> 9)
> >> at
> >> scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:59
> >> 7)
> >> at
> >> akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.s
> >> ca
> >> la:474) at
> >> akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.
> >> sc
> >> ala:425) at
> >> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:4
> >> 29
> >> ) at
> >> akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:381)
> >> at java.lang.Thread.run(Thread.java:745)
> >> 2017-06-20 16:45:13,990 INFO 
> >> org.apache.flink.yarn.YarnFlinkResourceManager>> 
> >>               - Trying to associate with JobManager leader
> >> 
> >> akka.tcp://flink@dmpkit-dev-dn2:24660/user/jobmanager 2017-06-20
> >> 16:45:13,993 WARN  akka.remote.ReliableDeliverySupervisor
> >> 
> >>     - Association with remote system
> >> 
> >> [akka.tcp://flink@dmpkit-dev-dn2:24660] has failed, address is now gated
> >> for [5000] ms. Reason: [Association failed with
> >> [akka.tcp://flink@dmpkit-dev-dn2:24660]] Caused by: [Connection refused:
> >> dmpkit-dev-dn2/136.243.170.205:24660] 2017-06-20 16:45:18,174 ERROR
> >> org.apache.flink.runtime.webmonitor.files.StaticFileServerHandler  -
> >> Caught
> >> exception akka.actor.ActorNotFound: Actor not found for:
> >> ActorSelection[Anchor(akka.tcp://flink@dmpkit-dev-dn2:24660/),
> >> Path(/user/jobmanager)] at
> >> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scal
> >> a:
> >> 65) at
> >> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scal
> >> a:
> >> 63) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
> >> 
> >> 	at
> >> 
> >> akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecuto
> >> r.
> >> scala:55) at
> >> akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:73) at
> >> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExec
> >> ut
> >> e(Future.scala:74) at
> >> akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:120)
> >> at
> >> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Futur
> >> e.
> >> scala:73) at
> >> scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
> >> at
> >> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:24
> >> 8)
> >> at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:266)
> >> 
> >> 	at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:533)
> >> 	at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:569)
> >> 	at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:559)
> >> 	at
> >> 
> >> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteA
> >> ct
> >> orRefProvider.scala:87) at
> >> akka.remote.EndpointWriter.postStop(Endpoint.scala:583)
> >> 
> >> 	at akka.actor.Actor$class.aroundPostStop(Actor.scala:477)
> >> 	at akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:437)
> >> 	at
> >> 
> >> akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$
> >> fi
> >> nishTerminate(FaultHandling.scala:210) at
> >> akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172)
> >> at akka.actor.ActorCell.terminate(ActorCell.scala:369)
> >> 
> >> 	at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462)
> >> 	at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
> >> 	at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
> >> 	at akka.dispatch.Mailbox.run(Mailbox.scala:219)
> >> 	at
> >> 
> >> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(Abstract
> >> Di
> >> spatcher.scala:397) at
> >> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at
> >> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.jav
> >> a:
> >> 1339) at
> >> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> >> at
> >> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.j
> >> av
> >> a:107) 2017-06-20 16:45:24,010 ERROR
> >> org.apache.flink.yarn.YarnFlinkResourceManager                - Resource
> >> manager could not register at JobManager
> >> akka.pattern.AskTimeoutException:
> >> Ask timed out on
> >> [ActorSelection[Anchor(akka.tcp://flink@dmpkit-dev-dn2:24660/),
> >> Path(/user/jobmanager)]] after [10000 ms] at
> >> akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:334
> >> )
> >> at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117)
> >> 
> >> 	at
> >> 
> >> scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future
> >> .s
> >> cala:599) at
> >> scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:10
> >> 9)
> >> at
> >> scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:59
> >> 7)
> >> at
> >> akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.s
> >> ca
> >> la:474) at
> >> akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.
> >> sc
> >> ala:425) at
> >> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:4
> >> 29
> >> ) at
> >> akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:381)
> >> at java.lang.Thread.run(Thread.java:745)
> >> 2017-06-20 16:45:24,010 INFO 
> >> org.apache.flink.yarn.YarnFlinkResourceManager>> 
> >>               - Trying to associate with JobManager leader
> >> 
> >> akka.tcp://flink@dmpkit-dev-dn2:24660/user/jobmanager 2017-06-20
> >> 16:45:24,014 WARN  akka.remote.ReliableDeliverySupervisor
> >> 
> >>     - Association with remote system
> >> 
> >> [akka.tcp://flink@dmpkit-dev-dn2:24660] has failed, address is now gated
> >> for [5000] ms. Reason: [Association failed with
> >> [akka.tcp://flink@dmpkit-dev-dn2:24660]] Caused by: [Connection refused:
> >> dmpkit-dev-dn2/136.243.170.205:24660] 2017-06-20 16:45:24,772 ERROR
> >> org.apache.flink.runtime.webmonitor.files.StaticFileServerHandler  -
> >> Caught
> >> exception akka.actor.ActorNotFound: Actor not found for:
> >> ActorSelection[Anchor(akka.tcp://flink@dmpkit-dev-dn2:24660/),
> >> Path(/user/jobmanager)] at
> >> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scal
> >> a:
> >> 65) at
> >> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scal
> >> a:
> >> 63) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
> >> 
> >> 	at
> >> 
> >> akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecuto
> >> r.
> >> scala:55) at
> >> akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:73) at
> >> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExec
> >> ut
> >> e(Future.scala:74) at
> >> akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:120)
> >> at
> >> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Futur
> >> e.
> >> scala:73) at
> >> scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
> >> at
> >> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:24
> >> 8)
> >> at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:266)
> >> 
> >> 	at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:533)
> >> 	at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:569)
> >> 	at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:559)
> >> 	at
> >> 
> >> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteA
> >> ct
> >> orRefProvider.scala:87) at
> >> akka.remote.EndpointWriter.postStop(Endpoint.scala:583)
> >> 
> >> 	at akka.actor.Actor$class.aroundPostStop(Actor.scala:477)
> >> 	at akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:437)
> >> 	at
> >> 
> >> akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$
> >> fi
> >> nishTerminate(FaultHandling.scala:210) at
> >> akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172)
> >> at akka.actor.ActorCell.terminate(ActorCell.scala:369)
> >> 
> >> 	at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462)
> >> 	at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
> >> 	at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
> >> 	at akka.dispatch.Mailbox.run(Mailbox.scala:219)
> >> 	at
> >> 
> >> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(Abstract
> >> Di
> >> spatcher.scala:397) at
> >> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at
> >> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.jav
> >> a:
> >> 1339) at
> >> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> >> at
> >> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.j
> >> av
> >> a:107) 2017-06-20 16:45:34,029 ERROR
> >> org.apache.flink.yarn.YarnFlinkResourceManager                - Resource
> >> manager could not register at JobManager
> >> akka.pattern.AskTimeoutException:
> >> Ask timed out on
> >> [ActorSelection[Anchor(akka.tcp://flink@dmpkit-dev-dn2:24660/),
> >> Path(/user/jobmanager)]] after [10000 ms] at
> >> akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:334
> >> )
> >> at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117)
> >> 
> >> 	at
> >> 
> >> scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future
> >> .s
> >> cala:599) at
> >> scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:10
> >> 9)
> >> at
> >> scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:59
> >> 7)
> >> at
> >> akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.s
> >> ca
> >> la:474) at
> >> akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.
> >> sc
> >> ala:425) at
> >> akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:4
> >> 29
> >> ) at
> >> akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:381)
> >> at java.lang.Thread.run(Thread.java:745)
> >> 2017-06-20 16:45:34,030 INFO 
> >> org.apache.flink.yarn.YarnFlinkResourceManager>> 
> >>               - Trying to associate with JobManager leader
> >> 
> >> akka.tcp://flink@dmpkit-dev-dn2:24660/user/jobmanager
> >> 
> >> 
> >> Finally, Job manager starts and works properly, just wanted to understand
> >> the cause of the error.
> >> 
> >> 
> >> Mant thanks in advance!
> >> 
> >> 
> >> Kind Regards,
> >> Mike Pryakhin
> >> 
> >>> On 20 Jun 2017, at 17:34, Nico Kruber <nico@data-artisans.com> wrote:
> >>> 
> >>> Hi Mike,
> >>> have you configured zookeeper [1] ? afaik, it is required for a high-
> >>> availability (YARN) session and is used to store JobManager state.
> >>> Without
> >>> it, a recovery would not know what to recover from.
> >>> 
> >>> 
> >>> Nico
> >>> 
> >>> [1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/
> >>> jobmanager_high_availability.html#yarn-cluster-high-availability
> >>> 
> >>> On Tuesday, 20 June 2017 13:23:35 CEST Mikhail Pryakhin wrote:
> >>>> Hello,
> >>>> 
> >>>> I'm currently trying to check whether my job is restarted in case of
> >>>> Job
> >>>> Manager failure. The job is submitted as a single job on YARN with the
> >>>> following options set in the flink-conf.yaml:
> >>>> 
> >>>> restart-strategy: fixed-delay
> >>>> restart-strategy.fixed-delay.attempts: 3
> >>>> restart-strategy.fixed-delay.delay: 10 s
> >>>> 
> >>>> Then I kill the Job Manager container. After that YARN starts a new
Job
> >>>> Manager container but the job is not started. What am I doing wrong?
Do
> >>>> I
> >>>> need something else to be configured to enable job restarts on JM
> >>>> failure?
> >>>> 
> >>>> I'm using flink 1.3 Hadoop 2.6
> >>>> 
> >>>> Thanks in advance.
> >>>> 
> >>>> Kind Regards,
> >>>> Mike Pryakhin


Mime
View raw message