hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Soni,Mohit" <mohitsoni1...@gmail.com>
Subject cgroups: Failed to initialize container executor
Date Fri, 30 May 2014 17:19:27 GMT
Hi
 
I am trying to use cgroups with YARN 2.4.0 single-node setup running on Ubuntu 13.10. It works
fine if I use DefaultContainerExecutor. But, I get error when I try to use LinuxContainerExecutor.
I referred to http://hadoop.apache.org/docs/r2.4.0/hadoop-project-dist/hadoop-common/SecureMode.html#LinuxContainerExecutor
for help.
 
My setup:
User: hduser
Group: hadoop

Permissions for bin directory:
hduser@host1:/usr/local/hadoop/bin$ ls -l
total 364
---Sr-s--- 1 root   hadoop  85670 May 30 00:14 container-executor
-r-------- 1 root   hadoop     97 May 30 00:52 container-executor.cfg
-rw-r--r-- 1 hduser hadoop     71 May 30 00:44 container-executor.cfg.orig
-rwxr-xr-x 1 hduser hadoop  85670 May 30 00:22 container-executor.orig
-rwxr-xr-x 1 hduser hadoop   4857 Mar 31 08:49 hadoop
-rwxr-xr-x 1 hduser hadoop   7541 Mar 31 08:49 hadoop.cmd
-rwxr-xr-x 1 hduser hadoop   8200 Mar 31 08:49 hdfs
-rwxr-xr-x 1 hduser hadoop   6162 Mar 31 08:49 hdfs.cmd
-rwxr-xr-x 1 hduser hadoop   5205 Mar 31 08:49 mapred
-rwxr-xr-x 1 hduser hadoop   5559 Mar 31 08:49 mapred.cmd
-rwxr-xr-x 1 hduser hadoop   1776 Mar 31 08:49 rcc
-rwxr-xr-x 1 hduser hadoop 108825 Mar 31 08:49 test-container-executor
-rwxr-xr-x 1 hduser hadoop   9539 Mar 31 08:49 yarn
-rwxr-xr-x 1 hduser hadoop   9988 Mar 31 08:49 yarn.cmd
 
File: container-executor.cfg
yarn.nodemanager.linux-container-executor.group=hadoop
min.user.id=1
allowed.system.users=hduser
 
File: yarn-site.xml
<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at
 
    http://www.apache.org/licenses/LICENSE-2.0
 
  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>
 
<!-- Site specific YARN configuration properties -->
 
<property>
   <name>yarn.nodemanager.aux-services</name>
   <value>mapreduce_shuffle</value>
</property>
<property>
   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
 
<!-- Cgroups -->
<property>
        <description>who will execute(launch) the containers.</description>
        <name>yarn.nodemanager.container-executor.class</name>
        <value>org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor</value>
</property>
<property>
        <description>The class which should help the LCE handle resources.</description>
        <name>yarn.nodemanager.linux-container-executor.resources-handler.class</name>
        <value>org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler</value>
</property>
<property>
        <name>yarn.nodemanager.linux-container-executor.cgroups.hierarchy</name>
        <value>/hadoop-yarn</value>
</property>
<property>
        <name>yarn.nodemanager.linux-container-executor.cgroups.mount</name>
        <value>true</value>
</property>
<property>
        <name>yarn.nodemanager.linux-container-executor.cgroups.mount-path</name>
        <value>/sys/fs/cgroup</value>
</property>
<property>
        <name>yarn.nodemanager.linux-container-executor.group</name>
        <value>hadoop</value>
</property>
 
Now, when I execute $HADOOP_INSTALL/bin/yarn nodemanager, I fails with exit code 255, with
following stacktrace:
 
14/05/30 01:10:10 INFO nodemanager.NodeManager: registered UNIX signal handlers for [TERM,
HUP, INT]
14/05/30 01:10:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
14/05/30 01:10:11 INFO service.AbstractService: Service NodeManager failed in state INITED;
cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container
executor
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:144)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:357)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:404)
Caused by: java.io.IOException: Cannot run program "/usr/local/hadoop/bin/container-executor":
error=2, No such file or directory
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1041)
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:448)
        at org.apache.hadoop.util.Shell.run(Shell.java:418)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
        at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:169)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:142)
        ... 3 more
Caused by: java.io.IOException: error=2, No such file or directory
        at java.lang.UNIXProcess.forkAndExec(Native Method)
        at java.lang.UNIXProcess.<init>(UNIXProcess.java:135)
        at java.lang.ProcessImpl.start(ProcessImpl.java:130)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1022)
        ... 8 more
14/05/30 01:10:11 FATAL nodemanager.NodeManager: Error starting NodeManager
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:144)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:357)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:404)
Caused by: java.io.IOException: Cannot run program "/usr/local/hadoop/bin/container-executor":
error=2, No such file or directory
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1041)
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:448)
        at org.apache.hadoop.util.Shell.run(Shell.java:418)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
        at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:169)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:142)
        ... 3 more
Caused by: java.io.IOException: error=2, No such file or directory
        at java.lang.UNIXProcess.forkAndExec(Native Method)
        at java.lang.UNIXProcess.<init>(UNIXProcess.java:135)
        at java.lang.ProcessImpl.start(ProcessImpl.java:130)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1022)
        ... 8 more
14/05/30 01:10:11 INFO nodemanager.NodeManager: SHUTDOWN_MSG:
 
It would be great if someone can please help me figure out what’s wrong with my setup.
 
Thanks
Mohit
Mime
View raw message