hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shane Kumpf (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-8485) Priviledged container app launch is failing intermittently
Date Mon, 02 Jul 2018 22:26:00 GMT

     [ https://issues.apache.org/jira/browse/YARN-8485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Shane Kumpf updated YARN-8485:
------------------------------
    Affects Version/s: 3.1.1
                       3.2.0
     Target Version/s: 3.2.0, 3.1.1
               Labels: Docker  (was: )

> Priviledged container app launch is failing intermittently
> ----------------------------------------------------------
>
>                 Key: YARN-8485
>                 URL: https://issues.apache.org/jira/browse/YARN-8485
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: yarn-native-services
>    Affects Versions: 3.2.0, 3.1.1
>         Environment: Debian
>            Reporter: Yesha Vora
>            Assignee: Eric Yang
>            Priority: Major
>              Labels: Docker
>         Attachments: YARN-8485.001.patch, YARN-8485.002.patch
>
>
> Privileged application fails intermittently 
> {code:java}
> yarn  jar /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell-*.jar 
-shell_command "sleep 30" -num_containers 1 -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker
-shell_env YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=xxx -shell_env YARN_CONTAINER_RUNTIME_DOCKER_RUN_PRIVILEGED_CONTAINER=true
-jar /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell-*.jar{code}
> Here,  container launch fails with 'Privileged containers are disabled' even though Docker
privilege container is enabled in the cluster
> {code:java|title=nm log}
> 2018-06-28 21:21:15,647 INFO  runtime.DockerLinuxContainerRuntime (DockerLinuxContainerRuntime.java:allowPrivilegedContainerExecution(664))
- All checks pass. Launching privileged container for : container_e01_1530220647587_0001_01_000002
> 2018-06-28 21:21:15,665 WARN  nodemanager.LinuxContainerExecutor (LinuxContainerExecutor.java:handleExitCode(593))
- Exit code from container container_e01_1530220647587_0001_01_000002 is : 29
> 2018-06-28 21:21:15,666 WARN  nodemanager.LinuxContainerExecutor (LinuxContainerExecutor.java:handleExitCode(599))
- Exception from container-launch with container ID: container_e01_1530220647587_0001_01_000002
and exit code: 29
> org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerExecutionException:
Launch container failed
>         at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(DockerLinuxContainerRuntime.java:958)
>         at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime.launchContainer(DelegatingLinuxContainerRuntime.java:141)
>         at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.handleLaunchForLaunchType(LinuxContainerExecutor.java:564)
>         at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:479)
>         at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:494)
>         at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:306)
>         at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:103)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> 2018-06-28 21:21:15,668 INFO  nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541))
- Exception from container-launch.
> 2018-06-28 21:21:15,668 INFO  nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541))
- Container id: container_e01_1530220647587_0001_01_000002
> 2018-06-28 21:21:15,668 INFO  nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541))
- Exit code: 29
> 2018-06-28 21:21:15,668 INFO  nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541))
- Exception message: Launch container failed
> 2018-06-28 21:21:15,668 INFO  nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541))
- Shell error output: check privileges failed for user: hrt_qa, error code: 0
> 2018-06-28 21:21:15,668 INFO  nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541))
- Privileged containers are disabled for user: hrt_qa
> 2018-06-28 21:21:15,668 INFO  nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541))
- Error constructing docker command, docker error code=11, error message='Privileged containers
are disabled'
> 2018-06-28 21:21:15,668 INFO  nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541))
-
> 2018-06-28 21:21:15,669 INFO  nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541))
- Shell output: main : command provided 4
> 2018-06-28 21:21:15,669 INFO  nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541))
- main : run as user is hrt_qa
> 2018-06-28 21:21:15,669 INFO  nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541))
- main : requested yarn user is hrt_qa
> 2018-06-28 21:21:15,669 INFO  nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541))
- Creating script paths...
> 2018-06-28 21:21:15,669 INFO  nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541))
- Creating local dirs...
> 2018-06-28 21:21:15,669 INFO  nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541))
- Creating script paths...
> 2018-06-28 21:21:15,669 INFO  nodemanager.ContainerExecutor (ContainerExecutor.java:logOutput(541))
- Creating local dirs...
> 2018-06-28 21:21:15,693 WARN  launcher.ContainerLaunch (ContainerLaunch.java:handleContainerExitWithFailure(598))
- Container launch failed : Container exited with a non-zero exit code 29.
> 2018-06-28 21:21:15,693 ERROR launcher.ContainerLaunch (ContainerLaunch.java:handleContainerExitWithFailure(623))
- Failed to get tail of the container's prelaunch error log file
> java.io.FileNotFoundException: File /grid/0/hadoop/yarn/log/application_1530220647587_0001/container_e01_1530220647587_0001_01_000002/prelaunch.err
does not exist
>         at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641)
>         at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:930)
>         at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
>         at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.handleContainerExitWithFailure(ContainerLaunch.java:609)
>         at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.handleContainerExitCode(ContainerLaunch.java:575)
>         at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:340)
>         at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:103)
 
>          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> 2018-06-28 21:21:15,704 INFO  container.ContainerImpl (ContainerImpl.java:handle(2093))
- Container container_e01_1530220647587_0001_01_000002 transitioned from RUNNING to EXITED_WITH_FAILURE{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message