hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "saurab (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-6790) Yarn : Exception from container-launch : Container failed with state: EXITED_WITH_FAILURE
Date Mon, 10 Jul 2017 08:47:00 GMT
saurab created YARN-6790:
----------------------------

             Summary: Yarn : Exception from container-launch : Container failed with state:
EXITED_WITH_FAILURE
                 Key: YARN-6790
                 URL: https://issues.apache.org/jira/browse/YARN-6790
             Project: Hadoop YARN
          Issue Type: Bug
         Environment: hadoop-2.8.0, tez-0.8.5
ram 8gb, Dell inspiron-15 3000 series intell-i5
            Reporter: saurab


I wanted to run hive queries through jdbc, but I am getting

java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code
1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
Then I looked nodemanager log. Here are some key notes to consider

1)Container container_1499666177243_0001_02_000001 transitioned from RUNNING to EXITED_WITH_FAILURERESULT=FAILURE
2)DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE
And here is complete stack trace

2017-07-10 11:41:34,149 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
Exception from container-launch with container ID: container_1499666177243_0001_02_000001
and exit code: 1
ExitCodeException exitCode=1: 
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:972)
    at org.apache.hadoop.util.Shell.run(Shell.java:869)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1170)
    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:236)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:305)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:84)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
2017-07-10 11:41:34,152 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
Exception from container-launch.
2017-07-10 11:41:34,152 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
Container id: container_1499666177243_0001_02_000001
2017-07-10 11:41:34,152 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
Exit code: 1
2017-07-10 11:41:34,152 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
Stack trace: ExitCodeException exitCode=1: 
2017-07-10 11:41:34,152 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
  at org.apache.hadoop.util.Shell.runCommand(Shell.java:972)
2017-07-10 11:41:34,152 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
  at org.apache.hadoop.util.Shell.run(Shell.java:869)
2017-07-10 11:41:34,152 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
  at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1170)
2017-07-10 11:41:34,152 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
  at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:236)
2017-07-10 11:41:34,152 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
  at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:305)
2017-07-10 11:41:34,152 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
  at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:84)
2017-07-10 11:41:34,152 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2017-07-10 11:41:34,152 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
2017-07-10 11:41:34,153 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2017-07-10 11:41:34,153 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
  at java.lang.Thread.run(Thread.java:748)
2017-07-10 11:41:34,153 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
Container exited with a non-zero exit code 1
2017-07-10 11:41:34,156 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Container container_1499666177243_0001_02_000001 transitioned from RUNNING to EXITED_WITH_FAILURE
2017-07-10 11:41:34,156 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
Cleaning up container container_1499666177243_0001_02_000001
2017-07-10 11:41:34,199 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=saurab
  OPERATION=Container Finished - Failed   TARGET=ContainerImpl    RESULT=FAILURE  DESCRIPTION=Container
failed with state: EXITED_WITH_FAILURE    APPID=application_1499666177243_0001    CONTAINERID=container_1499666177243_0001_02_000001
2017-07-10 11:41:34,200 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
Deleting absolute path : /home/saurab/hadoopec/hadoop/tmp/hadoop-tmp-dir/nm-local-dir/usercache/saurab/appcache/application_1499666177243_0001/container_1499666177243_0001_02_000001
2017-07-10 11:41:34,202 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Container container_1499666177243_0001_02_000001 transitioned from EXITED_WITH_FAILURE to
DONE
2017-07-10 11:41:34,203 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
Removing container_1499666177243_0001_02_000001 from application application_1499666177243_0001
2017-07-10 11:41:34,204 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Stopping resource-monitoring for container_1499666177243_0001_02_000001
2017-07-10 11:41:34,204 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
Got event CONTAINER_STOP for appId application_1499666177243_0001
2017-07-10 11:41:35,208 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
Removed completed containers from NM context: [container_1499666177243_0001_02_000001]
2017-07-10 11:41:35,209 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
Application application_1499666177243_0001 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP
2017-07-10 11:41:35,210 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
Deleting absolute path : /home/saurab/hadoopec/hadoop/tmp/hadoop-tmp-dir/nm-local-dir/usercache/saurab/appcache/application_1499666177243_0001
2017-07-10 11:41:35,210 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
Got event APPLICATION_STOP for appId application_1499666177243_0001
2017-07-10 11:41:35,211 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
Application application_1499666177243_0001 transitioned from APPLICATION_RESOURCES_CLEANINGUP
to FINISHED
2017-07-10 11:41:35,211 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler:
Scheduling Log Deletion for application: application_1499666177243_0001, with delay of 10800
seconds
2017-07-10 11:43:26,431 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful
for appattempt_1499666177243_0002_000002 (auth:SIMPLE)
2017-07-10 11:43:26,438 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Start request for container_1499666177243_0002_02_000001 by user saurab
2017-07-10 11:43:26,438 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Creating a new application reference for app application_1499666177243_0002
2017-07-10 11:43:26,439 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=saurab
  IP=10.10.10.149 OPERATION=Start Container Request   TARGET=ContainerManageImpl  RESULT=SUCCESS
 APPID=application_1499666177243_0002    CONTAINERID=container_1499666177243_0002_02_000001
2017-07-10 11:43:26,440 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
Application application_1499666177243_0002 transitioned from NEW to INITING
2017-07-10 11:43:26,440 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
Adding container_1499666177243_0002_02_000001 to application application_1499666177243_0002
2017-07-10 11:43:26,440 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
Application application_1499666177243_0002 transitioned from INITING to RUNNING
2017-07-10 11:43:26,441 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Container container_1499666177243_0002_02_000001 transitioned from NEW to LOCALIZING
2017-07-10 11:43:26,441 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
Got event CONTAINER_INIT for appId application_1499666177243_0002
2017-07-10 11:43:26,441 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
Got event APPLICATION_INIT for appId application_1499666177243_0002
2017-07-10 11:43:26,442 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
Got APPLICATION_INIT for service mapreduce_shuffle
2017-07-10 11:43:26,442 INFO org.apache.hadoop.mapred.ShuffleHandler: Added token for job_1499666177243_0002
2017-07-10 11:43:26,444 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
Resource hdfs://saurab:9000/tmp/hive/saurab/_tez_session_dir/fed51831-bf68-45b0-abea-11fb2b007c2f/.tez/application_1499666177243_0002/tez-conf.pb
transitioned from INIT to DOWNLOADING
2017-07-10 11:43:26,444 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
Resource hdfs://saurab:9000/tmp/hive/saurab/_tez_session_dir/fed51831-bf68-45b0-abea-11fb2b007c2f/.tez/application_1499666177243_0002/tez.session.local-resources.pb
transitioned from INIT to DOWNLOADING
2017-07-10 11:43:26,446 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Created localizer for container_1499666177243_0002_02_000001
2017-07-10 11:43:26,448 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Writing credentials to the nmPrivate file /home/saurab/hadoopec/hadoop/tmp/hadoop-tmp-dir/nm-local-dir/nmPrivate/container_1499666177243_0002_02_000001.tokens.
Credentials list: 
2017-07-10 11:43:26,449 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
Initializing user saurab
2017-07-10 11:43:26,450 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
Copying from /home/saurab/hadoopec/hadoop/tmp/hadoop-tmp-dir/nm-local-dir/nmPrivate/container_1499666177243_0002_02_000001.tokens
to /home/saurab/hadoopec/hadoop/tmp/hadoop-tmp-dir/nm-local-dir/usercache/saurab/appcache/application_1499666177243_0002/container_1499666177243_0002_02_000001.tokens
2017-07-10 11:43:26,450 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
Localizer CWD set to /home/saurab/hadoopec/hadoop/tmp/hadoop-tmp-dir/nm-local-dir/usercache/saurab/appcache/application_1499666177243_0002
= file:/home/saurab/hadoopec/hadoop/tmp/hadoop-tmp-dir/nm-local-dir/usercache/saurab/appcache/application_1499666177243_0002
2017-07-10 11:43:26,643 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
Resource hdfs://saurab:9000/tmp/hive/saurab/_tez_session_dir/fed51831-bf68-45b0-abea-11fb2b007c2f/.tez/application_1499666177243_0002/tez-conf.pb(->/home/saurab/hadoopec/hadoop/tmp/hadoop-tmp-dir/nm-local-dir/usercache/saurab/appcache/application_1499666177243_0002/filecache/10/tez-conf.pb)
transitioned from DOWNLOADING to LOCALIZED
2017-07-10 11:43:26,675 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
Resource hdfs://saurab:9000/tmp/hive/saurab/_tez_session_dir/fed51831-bf68-45b0-abea-11fb2b007c2f/.tez/application_1499666177243_0002/tez.session.local-resources.pb(->/home/saurab/hadoopec/hadoop/tmp/hadoop-tmp-dir/nm-local-dir/usercache/saurab/appcache/application_1499666177243_0002/filecache/11/tez.session.local-resources.pb)
transitioned from DOWNLOADING to LOCALIZED
2017-07-10 11:43:26,676 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Container container_1499666177243_0002_02_000001 transitioned from LOCALIZING to LOCALIZED
2017-07-10 11:43:26,715 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Container container_1499666177243_0002_02_000001 transitioned from LOCALIZED to RUNNING
2017-07-10 11:43:26,715 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Starting resource-monitoring for container_1499666177243_0002_02_000001
2017-07-10 11:43:26,718 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
launchContainer: [nice, -n, 0, bash, /home/saurab/hadoopec/hadoop/tmp/hadoop-tmp-dir/nm-local-dir/usercache/saurab/appcache/application_1499666177243_0002/container_1499666177243_0002_02_000001/default_container_executor.sh]
2017-07-10 11:43:26,868 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
Exit code from container container_1499666177243_0002_02_000001 is : 1
2017-07-10 11:43:26,868 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
Exception from container-launch with container ID: container_1499666177243_0002_02_000001
and exit code: 1
ExitCodeException exitCode=1: 
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:972)
    at org.apache.hadoop.util.Shell.run(Shell.java:869)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1170)
    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:236)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:305)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:84)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
2017-07-10 11:43:26,868 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
Exception from container-launch.
2017-07-10 11:43:26,868 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
Container id: container_1499666177243_0002_02_000001
2017-07-10 11:43:26,868 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
Exit code: 1
2017-07-10 11:43:26,868 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
Stack trace: ExitCodeException exitCode=1: 
2017-07-10 11:43:26,868 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
  at org.apache.hadoop.util.Shell.runCommand(Shell.java:972)
2017-07-10 11:43:26,868 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
  at org.apache.hadoop.util.Shell.run(Shell.java:869)
2017-07-10 11:43:26,868 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
  at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1170)
2017-07-10 11:43:26,868 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
  at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:236)
2017-07-10 11:43:26,868 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
  at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:305)
2017-07-10 11:43:26,868 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
  at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:84)
2017-07-10 11:43:26,868 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2017-07-10 11:43:26,868 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
2017-07-10 11:43:26,868 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2017-07-10 11:43:26,868 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
  at java.lang.Thread.run(Thread.java:748)
2017-07-10 11:43:26,868 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
Container exited with a non-zero exit code 1
2017-07-10 11:43:26,868 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Container container_1499666177243_0002_02_000001 transitioned from RUNNING to EXITED_WITH_FAILURE
2017-07-10 11:43:26,868 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
Cleaning up container container_1499666177243_0002_02_000001
2017-07-10 11:43:26,898 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
Deleting absolute path : /home/saurab/hadoopec/hadoop/tmp/hadoop-tmp-dir/nm-local-dir/usercache/saurab/appcache/application_1499666177243_0002/container_1499666177243_0002_02_000001
2017-07-10 11:43:26,899 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=saurab
  OPERATION=Container Finished - Failed   TARGET=ContainerImpl    RESULT=FAILURE  DESCRIPTION=Container
failed with state: EXITED_WITH_FAILURE    APPID=application_1499666177243_0002    CONTAINERID=container_1499666177243_0002_02_000001
2017-07-10 11:43:26,900 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Container container_1499666177243_0002_02_000001 transitioned from EXITED_WITH_FAILURE to
DONE
2017-07-10 11:43:26,900 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
Removing container_1499666177243_0002_02_000001 from application application_1499666177243_0002
2017-07-10 11:43:26,900 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Stopping resource-monitoring for container_1499666177243_0002_02_000001
2017-07-10 11:43:26,900 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
Got event CONTAINER_STOP for appId application_1499666177243_0002
2017-07-10 11:43:27,904 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
Removed completed containers from NM context: [container_1499666177243_0002_02_000001]
2017-07-10 11:43:27,905 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
Application application_1499666177243_0002 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP
2017-07-10 11:43:27,905 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
Deleting absolute path : /home/saurab/hadoopec/hadoop/tmp/hadoop-tmp-dir/nm-local-dir/usercache/saurab/appcache/application_1499666177243_0002
2017-07-10 11:43:27,905 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
Got event APPLICATION_STOP for appId application_1499666177243_0002
2017-07-10 11:43:27,905 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
Application application_1499666177243_0002 transitioned from APPLICATION_RESOURCES_CLEANINGUP
to FINISHED
2017-07-10 11:43:27,905 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler:
Scheduling Log Deletion for application: application_1499666177243_0002, with delay of 10800
seconds



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message