hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shuai Zhang <shu...@microsoft.com>
Subject [HELP] Failed to work with DockerContainerExecutor in Yarn
Date Fri, 22 Apr 2016 09:58:35 GMT
Hi there,

I’m trying to work with DockerContainerExecutor in Yarn, which is described in https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html

I followed the document and prepared everything, but failed to run even a simple wordcount
job with DockerContainerExecutor.

The console shows the failure:

> Diagnostics: Exception from container-launch: 
> ExitCodeException exitCode=126: Error: No such image or container: container_1461317429669_0001_02_000001
> Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt
dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
> bash: /tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1461317429669_0001/container_1461317429669_0001_02_000001/launch_container.sh:
Permission denied
>
>	at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
> 	at org.apache.hadoop.util.Shell.run(Shell.java:456)
> ... 
> Container exited with a non-zero exit code 126
> Failing this attempt. Failing the application.


The log of nodemanager shows:

> 2016-04-22 17:31:40,976 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Container container_1461317429669_0001_01_000001 transitioned from LOCALIZED to RUNNING
> 2016-04-22 17:31:40,992 DEBUG org.apache.hadoop.yarn.server.nodemanager.DockerContainerExecutor:
launchContainer: /usr/bin/docker run --rm --net=host  --name container_1461317429669_0001_01_000001
-v /tmp/hadoop-root/nm-local-dir:/tmp/hadoop-root/nm-local-dir -v /root/hadoop-2.7.2/logs/userlogs:/root/hadoop-2.7.2/logs/userlogs
-v /tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1461317429669_0001/container_1461317429669_0001_01_000001:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1461317429669_0001/container_1461317429669_0001_01_000001
sequenceiq/hadoop-docker:2.7.1 bash /tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1461317429669_0001/container_1461317429669_0001_01_000001/docker_container_executor.sh
> 2016-04-22 17:31:43,575 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Starting resource-monitoring for container_1461317429669_0001_01_000001
> 2016-04-22 17:31:45,832 WARN org.apache.hadoop.yarn.server.nodemanager.DockerContainerExecutor:
Exit code from container container_1461317429669_0001_01_000001 is : 126
> 2016-04-22 17:31:45,834 WARN org.apache.hadoop.yarn.server.nodemanager.DockerContainerExecutor:
Exception from container-launch with container ID: container_1461317429669_0001_01_000001
and exit code: 126
> ExitCodeException exitCode=126: Error: No such image or container: container_1461317429669_0001_01_000001
> Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt
dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
> bash: /tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1461317429669_0001/container_1461317429669_0001_01_000001/launch_container.sh:
Permission denied
>
>	at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
>	at org.apache.hadoop.util.Shell.run(Shell.java:456)
> ...
> 2016-04-22 17:31:45,836 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:

> 2016-04-22 17:31:45,837 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
Container exited with a non-zero exit code 126
> 2016-04-22 17:31:45,839 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Container container_1461317429669_0001_01_000001 transitioned from RUNNING to EXITED_WITH_FAILURE


I tried to run the launching command by manual, but got the same error: Permission denied.
But it differs when I append the "--privileged" options.

> /usr/bin/docker run --privileged --rm --net=host  --name container_1461317429669_0001_01_000001
-v /tmp/hadoop-root/nm-local-dir:/tmp/hadoop-root/nm-local-dir -v /root/hadoop-2.7.2/logs/userlogs:/root/hadoop-2.7.2/logs/userlogs
-v /tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1461317429669_0001/container_1461317429669_0001_01_000001:/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1461317429669_0001/container_1461317429669_0001_01_000001
sequenceiq/hadoop-docker:2.7.1 bash /tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1461317429669_0001/container_1461317429669_0001_01_000001/docker_container_executor.sh
> Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt
dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
> /tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1461317429669_0001/container_1461317429669_0001_01_000001/docker_container_executor_session.sh:
line 3: /usr/bin/docker: No such file or directory
> /tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1461317429669_0001/container_1461317429669_0001_01_000001/docker_container_executor_session.sh:
line 5: /usr/bin/docker: No such file or directory


Why the launching script need to run docker command within a docker container? Is there anybody
can show me a way to run jobs with DockerContainerExecutor?

Environment:
* CentOS 7
* Hadoop 2.7.2
* Docker 1.9.1
* Docker Image: sequenceiq/hadoop-docker:2.7.1

Reproduce steps:
* Download & install CentOS7 minimal ISO with single root user: http://isoredirect.centos.org/centos/7/isos/x86_64/CentOS-7-x86_64-Minimal-1511.iso
* yum install docker java-1.7.0-openjdk
* Download Hadoop 2.7.2: http://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.7.2/hadoop-2.7.2.tar.gz
* Prepare configuration files from here: https://onedrive.live.com/redir?resid=F41C57B4F6E1B6C9!1618&authkey=!AEvln9poXczhSX4&ithint=file%2cgz
* Setup JAVA_HOME & HADOOP_PREFIX
* docker pull sequenceiq/hadoop-docker:2.7.1
* $HADOOP_PREFIX/bin/hdfs namenode -format
* $HADOOP_PREFIX/sbin/start-all.sh
* $HADOOP_PREFIX/bin/hdfs dfs -mkdir -p /user/root
* $HADOOP_PREFIX/bin/hdfs dfs -put $HADOOP_PREFIX/LICENSE.txt
* $HADOOP_PREFIX/bin/hadoop jar $HADOOP_PREFIX/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar
wordcount -Dmapreduce.map.env="yarn.nodemanager.docker-container-executor.image-name=sequenceiq/hadoop-docker:2.7.1"
-Dyarn.app.mapreduce.am.env="yarn.nodemanager.docker-container-executor.image-name=sequenceiq/hadoop-docker:2.7.1"
LICENSE.txt wc_out

Additional info:
Related logs & configuration files & generated shell scripts are placed here: https://onedrive.live.com/redir?resid=F41C57B4F6E1B6C9!1621&authkey=!AGZttYYahRTc0KQ&ithint=folder%2cgz


Regards,
Shuai Zhang

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org
Mime
View raw message