hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 陈浩骏 <chen.hao...@immomo.com>
Subject Re: Unable to localize container, when using LinuxContainerExecutor
Date Tue, 07 Feb 2017 07:30:32 GMT
Thank you for your reply.

The problem is fixed.

Since the Hadoop version in use is 2.6.0, I diff the LinuxContainerExecutor.java to the most
updated one.
And I replace ‘,’ to ‘%’ in startLocalizer(…) and launchContainer(…) method respectively,
like the following code block:
“
command.addAll(Arrays.asList(containerExecutorExe,
                   runAsUser,
                   user,
                   Integer.toString(Commands.INITIALIZE_CONTAINER.getValue()),
                   appId,
                   nmPrivateContainerTokensPath.toUri().getPath().toString(),
                   StringUtils.join(",", localDirs),  -->           StringUtils.join("%",
localDirs),
                   StringUtils.join(",", logDirs)));  -->  StringUtils.join("%", logDirs)));

”.
The mr pi finishes.

It seems that the ‘,’ would fail the execution of running container-executor.

Regards,
Reid Chan

From: 萝卜丝炒饭 <1427357147@qq.com>
Date: Tuesday, 7 February 2017 at 2:31 PM
To: 陈浩骏 <chen.haojun@immomo.com>, "user@hadoop.apache.org" <user@hadoop.apache.org>
Subject: Re: Unable to localize container, when using LinuxContainerExecutor

have you created the parent folder, such as  /data1/yarn
---Original---
From: "陈浩骏"<chen.haojun@immomo.com>
Date: 2017/2/7 00:43:47
To: "user@hadoop.apache.org"<user@hadoop.apache.org>;
Subject: Unable to localize container, when using LinuxContainerExecutor

Hi,

   I tried to run secure containers in cluster, and followed the instructions from http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/SecureMode.html#LinuxContainerExecutor
   After that, I ran the mapreduce example pi and got the following exception:

2017-02-06 17:46:24,018 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
main : command provided 0
2017-02-06 17:46:24,018 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
main : run as user is hadoop
2017-02-06 17:46:24,018 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
main : requested yarn user is hadoop
2017-02-06 17:46:24,019 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
Failed to create directory /data1/yarn/nm,/data2/yarn/nm,/data3/yarn/nm/usercache/hadoop -
No such file or directory
2017-02-06 17:46:24,019 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Localizer failed
java.io.IOException: Application application_1486374303689_0001 initialization failed (exitCode=20)
with output: main : command provided 0
main : run as user is hadoop
main : requested yarn user is hadoop
Failed to create directory /data1/yarn/nm,/data2/yarn/nm,/data3/yarn/nm/usercache/hadoop -
No such file or directory

        at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.startLocalizer(LinuxContainerExecutor.java:241)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1088)
Caused by: ExitCodeException exitCode=20:
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
        at org.apache.hadoop.util.Shell.run(Shell.java:455)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
        at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.startLocalizer(LinuxContainerExecutor.java:232)
        ... 1 more


  Here is the container-executor.cfg under /etc/hadoop owned by root (configurations are same
to the yarn-site.xml):
yarn.nodemanager.local-dirs=/data1/yarn/nm,/data2/yarn/nm,/data3/yarn/nm
yarn.nodemanager.log-dirs=/data1/hadoop/logs/container
yarn.nodemanager.linux-container-executor.group=hadoop
banned.users=yarn
min.user.id=500
allowed.system.users=hadoop,dm

    Hope to get some helps.

    Regards,
    Reid Chan.
Mime
View raw message