hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sam liu <samliuhad...@gmail.com>
Subject Re: How to use docker in Hadoop, with patch of YARN-1964?
Date Wed, 13 Aug 2014 03:13:53 GMT
After applying this patch, I added following config in yarn-site.xml:
  <property>
    <name>yarn.nodemanager.container-executor.class</name>

<value>org.apache.hadoop.yarn.server.nodemanager.DockerContainerExecutor</value>
  </property>

Then I can start NodeManager with enabling DockerContainerExecutor. But
failed to execute a simple mr job, and the exception is* 'Cannot connect to
the Docker daemon. Is 'docker -d' running on this host?'*. Below are the
full exception info and any body could give me some hints?

14/08/12 20:06:14 INFO mapreduce.Job: Job job_1407899030909_0002 running in
uber mode : false
14/08/12 20:06:14 INFO mapreduce.Job:  map 0% reduce 0%
14/08/12 20:06:14 INFO mapreduce.Job: Job job_1407899030909_0002 failed
with state FAILED due to: Application application_1407899030909_0002
failed 2 times due to AM Container for appattempt_1407899030909_0002_000002
exited with  exitCode: 1 due to: Exception from container-launc    h:
org.apache.hadoop.util.Shell$ExitCodeException: Warning: '-rm' is
deprecated, it will be replaced by '--rm' soon. See usage.
Warning: '-name' is deprecated, it will be replaced by '--name' soon. See
usage.
2014/08/12 20:06:13 Cannot connect to the Docker daemon. Is 'docker -d'
running on this host?

        at org.apache.hadoop.util.Shell.runCommand(Shell.java:444)
        at org.apache.hadoop.util.Shell.run(Shell.java:359)
        at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:569)
        at
org.apache.hadoop.yarn.server.nodemanager.DockerContainerExecutor.launchContainer(DockerContainerExecutor.java:174)
        at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:319)
        at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
        at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314)
        at java.util.concurrent.FutureTask.run(FutureTask.java:149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
        at java.lang.Thread.run(Thread.java:738)

Thanks!


2014-08-12 2:18 GMT-07:00 sam liu <samliuhadoop@gmail.com>:

> Hi Experts,
>
> I am very interesting that Hadoop could work with Docker and doing some
> trial on patch of YARN-1964.
>
> I applied patch yarn-1964-branch-2.2.0-docker.patch of jira YARN-1964 on
> branch 2.2 and am going to install a Hadoop cluster using the new generated
> tarball including the patch.
>
> Then, I think I can use DockerContainerExecutor, but I do not know much
> details on the usage and have following questions:
>
>
>
> *1. After installation, What's the detailed config steps to adopt
> DockerContainerExecutor? *
>
>
> *2. How to verify whether a MR task is really launched in Docker container
> not Yarn container?*
> *3. Which hadoop branch will officially include Docker support?*
>
> Thanks a lot!
>

Mime
View raw message