flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anchit Jatana <development.anc...@gmail.com>
Subject Flink Application on YARN failed on losing Job Manager | No recovery | Need help debug the cause from logs
Date Wed, 02 Nov 2016 17:44:54 GMT
Hi All,

I started my flink application on YARN using flink run -m yarn-cluster,
after running smoothly for 20 hrs it failed. Ideally the application should
have recover on losing the Job Manger (which runs in the same container as
the application master) pertaining to the fault tolerant nature of flink on
YARN but it didn't recover and failed. 

Please help me debug the logs. 

Thank you

Regards,
Anchit

Below are the logs:

2016-11-01 14:12:37,592 INFO  org.apache.flink.runtime.client.JobClientActor             
 
- 11/01/2016 14:12:36	Parse & Map Record - (Visitor ID, Product List)  ->
Filtering None Objects -> Fetching Output(148/200) switched to RUNNING 
2016-11-02 10:16:42,960 INFO 
org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider  - Failing
over to rm1
2016-11-02 10:17:24,026 INFO 
org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider  - Failing
over to rm2
2016-11-02 10:17:40,882 INFO 
org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider  - Failing
over to rm1
2016-11-02 10:24:41,964 WARN  akka.remote.ReliableDeliverySupervisor                     
 
- Association with remote system [akka.tcp://flink@10.66.245.26:47722] has
failed, address is now gated for [5000] ms. Reason is: [Disassociated].
2016-11-02 10:24:56,311 WARN  Remoting                                                   
 
- Tried to associate with unreachable remote address
[akka.tcp://flink@10.66.245.26:47722]. Address is now gated for 5000 ms, all
messages to this address will be delivered to dead letters. Reason:
Connection refused: /10.66.245.26:47722
2016-11-02 10:24:56,315 INFO  org.apache.flink.runtime.client.JobClientActor             
 
- Lost connection to JobManager
akka.tcp://flink@10.66.245.26:47722/user/jobmanager. Triggering connection
timeout.
2016-11-02 10:24:56,315 INFO  org.apache.flink.runtime.client.JobClientActor             
 
- Disconnect from JobManager
Actor[akka.tcp://flink@10.66.245.26:47722/user/jobmanager#1251121709].
2016-11-02 10:25:56,330 INFO  org.apache.flink.runtime.client.JobClientActor             
 
- Terminate JobClientActor.
2016-11-02 10:25:56,331 INFO  org.apache.flink.runtime.client.JobClientActor             
 
- Disconnect from JobManager null.
2016-11-02 10:25:56,333 ERROR org.apache.flink.client.CliFrontend                        
 
- Error while running the command.
org.apache.flink.client.program.ProgramInvocationException: The program
execution failed: Communication with JobManager failed: Lost connection to
the JobManager.
	at
org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:405)
	at
org.apache.flink.yarn.YarnClusterClient.submitJob(YarnClusterClient.java:204)
	at
org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:378)
	at
org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:68)
	at
org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:585)
	at
com.tgt.prz.streaming.recs.drivers.SessionRecs2$.main(SessionRecs2.scala:126)
	at com.tgt.prz.streaming.recs.drivers.SessionRecs2.main(SessionRecs2.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:509)
	at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403)
	at
org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:320)
	at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:777)
	at org.apache.flink.client.CliFrontend.run(CliFrontend.java:253)
	at org.apache.flink.client.CliFrontend$2.run(CliFrontend.java:997)
	at org.apache.flink.client.CliFrontend$2.run(CliFrontend.java:994)
	at
org.apache.flink.runtime.security.SecurityUtils$1.run(SecurityUtils.java:56)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
	at
org.apache.flink.runtime.security.SecurityUtils.runSecured(SecurityUtils.java:53)
	at
org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:994)
	at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1048)
Caused by: org.apache.flink.runtime.client.JobExecutionException:
Communication with JobManager failed: Lost connection to the JobManager.
	at
org.apache.flink.runtime.client.JobClient.submitJobAndWait(JobClient.java:137)
	at
org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:401)
	... 24 more
Caused by:
org.apache.flink.runtime.client.JobClientActorConnectionTimeoutException:
Lost connection to the JobManager.
	at
org.apache.flink.runtime.client.JobClientActor.handleMessage(JobClientActor.java:252)
	at
org.apache.flink.runtime.akka.FlinkUntypedActor.handleLeaderSessionID(FlinkUntypedActor.java:90)
	at
org.apache.flink.runtime.akka.FlinkUntypedActor.onReceive(FlinkUntypedActor.java:70)
	at
akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
	at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
	at akka.actor.ActorCell.invoke(ActorCell.scala:487)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
	at akka.dispatch.Mailbox.run(Mailbox.scala:221)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
	at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2016-11-02 10:25:56,341 INFO  org.apache.flink.yarn.YarnClusterClient                    
 
- Sending shutdown request to the Application Master
2016-11-02 10:25:56,341 INFO  org.apache.flink.yarn.YarnClusterClient                    
 
- Start application client.
2016-11-02 10:25:56,344 WARN  org.apache.flink.yarn.YarnClusterClient                    
 
- YARN reported application state FAILED
2016-11-02 10:25:56,344 WARN  org.apache.flink.yarn.YarnClusterClient                    
 
- Diagnostics: Application application_1476277440022_40328 failed 1 times
due to Attempt recovered after RM restartAM Container for
appattempt_1476277440022_40328_000001 exited with  exitCode: 243
For more detailed output, check application tracking
page:http://d-3zkvk02.target.com:8088/cluster/app/application_1476277440022_40328Then,
click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_e3066_1476277440022_40328_01_000001
Exit code: 243
Stack trace: ExitCodeException exitCode=243: 
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:576)
	at org.apache.hadoop.util.Shell.run(Shell.java:487)
	at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753)
	at
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:371)
	at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
	at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Shell output: main : command provided 1
main : run as user is A12345
main : requested yarn user is A12345


Container exited with a non-zero exit code 243
Failing this attempt. Failing the application.
2016-11-02 10:25:56,346 INFO  org.apache.flink.yarn.ApplicationClient                    
 
- Notification about new leader address
akka.tcp://flink@10.66.245.26:47722/user/jobmanager with session ID null.
2016-11-02 10:25:56,349 INFO  org.apache.flink.yarn.ApplicationClient                    
 
- Sending StopCluster request to JobManager.
2016-11-02 10:25:56,350 INFO  org.apache.flink.yarn.ApplicationClient                    
 
- Received address of new leader
akka.tcp://flink@10.66.245.26:47722/user/jobmanager with session ID null.
2016-11-02 10:25:56,351 INFO  org.apache.flink.yarn.ApplicationClient                    
 
- Disconnect from JobManager null.
2016-11-02 10:25:56,353 INFO  org.apache.flink.yarn.ApplicationClient                    
 
- Trying to register at JobManager
akka.tcp://flink@10.66.245.26:47722/user/jobmanager.
2016-11-02 10:25:56,363 WARN  Remoting                                                   
 
- Tried to associate with unreachable remote address
[akka.tcp://flink@10.66.245.26:47722]. Address is now gated for 5000 ms, all
messages to this address will be delivered to dead letters. Reason:
Connection refused: /10.66.245.26:47722
2016-11-02 10:25:56,870 INFO  org.apache.flink.yarn.ApplicationClient                    
 
- Trying to register at JobManager
akka.tcp://flink@10.66.245.26:47722/user/jobmanager.
2016-11-02 10:25:57,369 INFO  org.apache.flink.yarn.ApplicationClient                    
 
- Sending StopCluster request to JobManager.
2016-11-02 10:25:57,889 INFO  org.apache.flink.yarn.ApplicationClient                    
 
- Trying to register at JobManager
akka.tcp://flink@10.66.245.26:47722/user/jobmanager.
2016-11-02 10:25:58,389 INFO  org.apache.flink.yarn.ApplicationClient                    
 
- Sending StopCluster request to JobManager.
2016-11-02 10:25:59,410 INFO  org.apache.flink.yarn.ApplicationClient                    
 
- Sending StopCluster request to JobManager.
2016-11-02 10:25:59,909 INFO  org.apache.flink.yarn.ApplicationClient                    
 
- Trying to register at JobManager
akka.tcp://flink@10.66.245.26:47722/user/jobmanager.
2016-11-02 10:26:00,429 INFO  org.apache.flink.yarn.ApplicationClient                    
 
- Sending StopCluster request to JobManager.
2016-11-02 10:26:01,449 INFO  org.apache.flink.yarn.ApplicationClient                    
 
- Sending StopCluster request to JobManager.
2016-11-02 10:26:02,469 INFO  org.apache.flink.yarn.ApplicationClient                    
 
- Sending StopCluster request to JobManager.
2016-11-02 10:26:03,489 INFO  org.apache.flink.yarn.ApplicationClient                    
 
- Sending StopCluster request to JobManager.
2016-11-02 10:26:03,929 INFO  org.apache.flink.yarn.ApplicationClient                    
 
- Trying to register at JobManager
akka.tcp://flink@10.66.245.26:47722/user/jobmanager.
2016-11-02 10:26:03,935 WARN  Remoting                                                   
 
- Tried to associate with unreachable remote address
[akka.tcp://flink@10.66.245.26:47722]. Address is now gated for 5000 ms, all
messages to this address will be delivered to dead letters. Reason:
Connection refused: /10.66.245.26:47722
2016-11-02 10:26:04,509 INFO  org.apache.flink.yarn.ApplicationClient                    
 
- Sending StopCluster request to JobManager.
2016-11-02 10:26:05,529 INFO  org.apache.flink.yarn.ApplicationClient                    
 
- Sending StopCluster request to JobManager.
2016-11-02 10:26:06,345 WARN  org.apache.flink.yarn.YarnClusterClient                    
 
- Error while stopping YARN cluster.
java.util.concurrent.TimeoutException: Futures timed out after [10000
milliseconds]
	at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
	at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:153)
	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
	at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:86)
	at
scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
	at scala.concurrent.Await$.ready(package.scala:86)
	at scala.concurrent.Await.ready(package.scala)
	at
org.apache.flink.yarn.YarnClusterClient.shutdownCluster(YarnClusterClient.java:366)
	at
org.apache.flink.yarn.YarnClusterClient.finalizeCluster(YarnClusterClient.java:336)
	at
org.apache.flink.client.program.ClusterClient.shutdown(ClusterClient.java:206)
	at org.apache.flink.client.CliFrontend.run(CliFrontend.java:260)
	at org.apache.flink.client.CliFrontend$2.run(CliFrontend.java:997)
	at org.apache.flink.client.CliFrontend$2.run(CliFrontend.java:994)
	at
org.apache.flink.runtime.security.SecurityUtils$1.run(SecurityUtils.java:56)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
	at
org.apache.flink.runtime.security.SecurityUtils.runSecured(SecurityUtils.java:53)
	at
org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:994)
	at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1048)
2016-11-02 10:26:06,347 INFO  org.apache.flink.yarn.YarnClusterClient                    
 
- Deleting files in
hdfs://littleredns/user/A12345/.flink/application_1476277440022_40328
2016-11-02 10:26:06,530 INFO  org.apache.flink.yarn.YarnClusterClient                    
 
- Application application_1476277440022_40328 finished with state FAILED and
final state FAILED at 1478100282775
2016-11-02 10:26:06,530 WARN  org.apache.flink.yarn.YarnClusterClient                    
 
- Application failed. Diagnostics Application
application_1476277440022_40328 failed 1 times due to Attempt recovered
after RM restartAM Container for appattempt_1476277440022_40328_000001
exited with  exitCode: 243
For more detailed output, check application tracking
page:http://d-3zkvk02.target.com:8088/cluster/app/application_1476277440022_40328Then,
click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_e3066_1476277440022_40328_01_000001
Exit code: 243
Stack trace: ExitCodeException exitCode=243: 
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:576)
	at org.apache.hadoop.util.Shell.run(Shell.java:487)
	at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753)
	at
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:371)
	at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
	at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Shell output: main : command provided 1
main : run as user is A12345
main : requested yarn user is A12345


Container exited with a non-zero exit code 243
Failing this attempt. Failing the application.
2016-11-02 10:26:06,531 WARN  org.apache.flink.yarn.YarnClusterClient                    
 
- If log aggregation is activated in the Hadoop cluster, we recommend to
retrieve the full application log using this command:
	yarn logs -appReport application_1476277440022_40328
(It sometimes takes a few seconds until the logs are aggregated)
2016-11-02 10:26:06,531 INFO  org.apache.flink.yarn.YarnClusterClient                    
 
- YARN Client is shutting down
2016-11-02 10:26:06,532 INFO  org.apache.flink.yarn.ApplicationClient                    
 
- Stopped Application client.
2016-11-02 10:26:06,533 INFO  org.apache.flink.yarn.ApplicationClient                    
 
- Disconnect from JobManager null.
2016-11-02 10:26:06,536 INFO 
akka.remote.RemoteActorRefProvider$RemotingTerminator         - Shutting
down remote daemon.
2016-11-02 10:26:06,537 INFO 
akka.remote.RemoteActorRefProvider$RemotingTerminator         - Remote
daemon shut down; proceeding with flushing remote transports.
2016-11-02 10:26:06,558 INFO 
akka.remote.RemoteActorRefProvider$RemotingTerminator         - Remoting
shut down.






--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Application-on-YARN-failed-on-losing-Job-Manager-No-recovery-Need-help-debug-the-cause-from-los-tp9839.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Mime
View raw message