hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-8064) Docker ".cmd" files should not be put in hadoop.tmp.dir
Date Thu, 19 Apr 2018 22:50:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-8064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16444931#comment-16444931
] 

Eric Yang commented on YARN-8064:
---------------------------------

[~ebadger] I am getting errors when trying out the patch 008:

{code}
Invalid conf file provided, unable to open file : /tmp/hadoop-yarn/nm-local-dir/nmPrivate/application_1524177007026_0001/container_1524177007026_0001_01_000003/docker.container_1524177007026_0001_01_0000031851872809113698536.cmd
Error constructing docker command, docker error code=1, error message='Invalid command file
passed'

Stdout: main : command provided 4
main : run as user is hbase
main : requested yarn user is hbase
Creating script paths...
Creating local dirs...

Full command array for failed execution:
[/usr/local/hadoop-3.2.0-SNAPSHOT/bin/container-executor, hbase, hbase, 4, application_1524177007026_0001,
container_1524177007026_0001_01_000003, /tmp/hadoop-yarn/nm-local-dir/usercache/hbase/appcache/application_1524177007026_0001/container_1524177007026_0001_01_000003,
/tmp/hadoop-yarn/nm-local-dir/nmPrivate/application_1524177007026_0001/container_1524177007026_0001_01_000003/launch_container.sh,
/tmp/hadoop-yarn/nm-local-dir/nmPrivate/application_1524177007026_0001/container_1524177007026_0001_01_000003/container_1524177007026_0001_01_000003.tokens,
/tmp/hadoop-yarn/nm-local-dir/nmPrivate/application_1524177007026_0001/container_1524177007026_0001_01_000003/container_1524177007026_0001_01_000003.pid,
/tmp/hadoop-yarn/nm-local-dir, /usr/local/hadoop-3.2.0-SNAPSHOT/logs/userlogs, /tmp/hadoop-yarn/nm-local-dir/nmPrivate/application_1524177007026_0001/container_1524177007026_0001_01_000003/docker.container_1524177007026_0001_01_0000031851872809113698536.cmd,
cgroups=none]
2018-04-19 22:31:31,093 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime:
Launch container failed. Exception:
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException:
ExitCodeException exitCode=29: Invalid conf file provided, unable to open file : /tmp/hadoop-yarn/nm-local-dir/nmPrivate/application_1524177007026_0001/container_1524177007026_0001_01_000003/docker.container_1524177007026_0001_01_0000031851872809113698536.cmd
Error constructing docker command, docker error code=1, error message='Invalid command file
passed'

        at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:180)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(DockerLinuxContainerRuntime.java:910)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime.launchContainer(DelegatingLinuxContainerRuntime.java:141)
        at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.handleLaunchForLaunchType(LinuxContainerExecutor.java:564)
        at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:479)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:492)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:304)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:101)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: ExitCodeException exitCode=29: Invalid conf file provided, unable to open file
: /tmp/hadoop-yarn/nm-local-dir/nmPrivate/application_1524177007026_0001/container_1524177007026_0001_01_000003/docker.container_1524177007026_0001_01_0000031851872809113698536.cmd
Error constructing docker command, docker error code=1, error message='Invalid command file
passed'

        at org.apache.hadoop.util.Shell.runCommand(Shell.java:1009)
        at org.apache.hadoop.util.Shell.run(Shell.java:902)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1227)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:152)
        ... 11 more
{code}

After the first occurrence of logged exception happens, I don't see cmd files created for
other containers.


> Docker ".cmd" files should not be put in hadoop.tmp.dir
> -------------------------------------------------------
>
>                 Key: YARN-8064
>                 URL: https://issues.apache.org/jira/browse/YARN-8064
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Eric Badger
>            Assignee: Eric Badger
>            Priority: Critical
>         Attachments: YARN-8064.001.patch, YARN-8064.002.patch, YARN-8064.003.patch, YARN-8064.004.patch,
YARN-8064.005.patch, YARN-8064.006.patch, YARN-8064.007.patch, YARN-8064.008.patch
>
>
> Currently all of the docker command files are being put into {{hadoop.tmp.dir}}, which
doesn't get cleaned up. So, eventually all of the inodes will fill up and no more tasks will
be able to run



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message