Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DDDAA1106F for ; Wed, 13 Aug 2014 03:14:27 +0000 (UTC) Received: (qmail 45759 invoked by uid 500); 13 Aug 2014 03:14:21 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 45622 invoked by uid 500); 13 Aug 2014 03:14:21 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 45612 invoked by uid 99); 13 Aug 2014 03:14:21 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Aug 2014 03:14:21 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of samliuhadoop@gmail.com designates 209.85.220.172 as permitted sender) Received: from [209.85.220.172] (HELO mail-vc0-f172.google.com) (209.85.220.172) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Aug 2014 03:13:54 +0000 Received: by mail-vc0-f172.google.com with SMTP id im17so14429350vcb.17 for ; Tue, 12 Aug 2014 20:13:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=QAMVJeQYQks+BHBlV8aIqixZMmtxgxMUnQsIGrL04m0=; b=C8fgLhND5DrC1W+Ehc3TGeuTuN0clMRgHXSdFWADQJ4idrN74Yvbu6Y6hXjjEOIKqz /Dq7zWSWtaJFCQ9iSgmDY+nj1HxQDagCge8YzG5XiHLS+1V1naOeJ5WWMHkfaowSVCMs pkQLSezAlm1XzUH3LtpQzPWfQdJwOuOX6JiOtywGtGWoRF+XCwqRtEoIVCx8WNUltVMv yI7WGucGKMFGecnSNBkPOWduy75aVvucd36BZoTtcDoFtUNzMG/cKupwetL3Pdmp7LBu Zkim1aR0BnhoncZRgTu+iOAkS4jykjg5ErHsqoHC9rtXbJMeQabSV/J9lUqT2Vx7ETbr 3wRQ== MIME-Version: 1.0 X-Received: by 10.52.245.66 with SMTP id xm2mr1142398vdc.36.1407899633184; Tue, 12 Aug 2014 20:13:53 -0700 (PDT) Received: by 10.220.150.202 with HTTP; Tue, 12 Aug 2014 20:13:53 -0700 (PDT) In-Reply-To: References: Date: Tue, 12 Aug 2014 20:13:53 -0700 Message-ID: Subject: Re: How to use docker in Hadoop, with patch of YARN-1964? From: sam liu To: "user@hadoop.apache.org" Content-Type: multipart/alternative; boundary=001a11c257aa83f4cb05007a2d84 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c257aa83f4cb05007a2d84 Content-Type: text/plain; charset=UTF-8 After applying this patch, I added following config in yarn-site.xml: yarn.nodemanager.container-executor.class org.apache.hadoop.yarn.server.nodemanager.DockerContainerExecutor Then I can start NodeManager with enabling DockerContainerExecutor. But failed to execute a simple mr job, and the exception is* 'Cannot connect to the Docker daemon. Is 'docker -d' running on this host?'*. Below are the full exception info and any body could give me some hints? 14/08/12 20:06:14 INFO mapreduce.Job: Job job_1407899030909_0002 running in uber mode : false 14/08/12 20:06:14 INFO mapreduce.Job: map 0% reduce 0% 14/08/12 20:06:14 INFO mapreduce.Job: Job job_1407899030909_0002 failed with state FAILED due to: Application application_1407899030909_0002 failed 2 times due to AM Container for appattempt_1407899030909_0002_000002 exited with exitCode: 1 due to: Exception from container-launc h: org.apache.hadoop.util.Shell$ExitCodeException: Warning: '-rm' is deprecated, it will be replaced by '--rm' soon. See usage. Warning: '-name' is deprecated, it will be replaced by '--name' soon. See usage. 2014/08/12 20:06:13 Cannot connect to the Docker daemon. Is 'docker -d' running on this host? at org.apache.hadoop.util.Shell.runCommand(Shell.java:444) at org.apache.hadoop.util.Shell.run(Shell.java:359) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:569) at org.apache.hadoop.yarn.server.nodemanager.DockerContainerExecutor.launchContainer(DockerContainerExecutor.java:174) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:319) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314) at java.util.concurrent.FutureTask.run(FutureTask.java:149) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919) at java.lang.Thread.run(Thread.java:738) Thanks! 2014-08-12 2:18 GMT-07:00 sam liu : > Hi Experts, > > I am very interesting that Hadoop could work with Docker and doing some > trial on patch of YARN-1964. > > I applied patch yarn-1964-branch-2.2.0-docker.patch of jira YARN-1964 on > branch 2.2 and am going to install a Hadoop cluster using the new generated > tarball including the patch. > > Then, I think I can use DockerContainerExecutor, but I do not know much > details on the usage and have following questions: > > > > *1. After installation, What's the detailed config steps to adopt > DockerContainerExecutor? * > > > *2. How to verify whether a MR task is really launched in Docker container > not Yarn container?* > *3. Which hadoop branch will officially include Docker support?* > > Thanks a lot! > --001a11c257aa83f4cb05007a2d84 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
After applying this patch, I added following config i= n yarn-site.xml:
=C2=A0 <property>
=C2=A0=C2=A0=C2=A0 <name&= gt;yarn.nodemanager.container-executor.class</name>
=C2=A0=C2=A0 &= lt;value>org.apache.hadoop.yarn.server.nodemanager.DockerContainerExecut= or</value>
=C2=A0 </property>

Then I can start NodeManager with enabling = DockerContainerExecutor. But failed to execute a simple mr job, and the exc= eption is 'Cannot connect to the Docker daemon. Is 'docker -d= 9; running on this host?'. Below are the full exception info and an= y body could give me some hints?

14/08/12 20:06:14 INFO mapreduce.Job: Job job_1407899030909_0002 runnin= g in uber mode : false
14/08/12 20:06:14 INFO mapreduce.Job:=C2=A0 map 0= % reduce 0%
14/08/12 20:06:14 INFO mapreduce.Job: Job job_1407899030909_= 0002 failed with state FAILED due to: Application application_1407899030909= _0002=C2=A0=C2=A0=C2=A0=C2=A0 failed 2 times due to AM Container for appatt= empt_1407899030909_0002_000002 exited with=C2=A0 exitCode: 1 due to: Except= ion from container-launc=C2=A0=C2=A0=C2=A0 h:
org.apache.hadoop.util.Shell$ExitCodeException: Warning: '-rm' is d= eprecated, it will be replaced by '--rm' soon. See usage.
Warnin= g: '-name' is deprecated, it will be replaced by '--name' s= oon. See usage.
2014/08/12 20:06:13 Cannot connect to the Docker daemon. Is 'docker -d&= #39; running on this host?

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 at org.apache.hadoop.util.Shell.runCommand(Shell.java:444)
=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.util.Shell.run(Shell= .java:359)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.util.Shell$= ShellCommandExecutor.execute(Shell.java:569)
=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 at org.apache.hadoop.yarn.server.nodemanager.DockerContaine= rExecutor.launchContainer(DockerContainerExecutor.java:174)
=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.yarn.server.nodemanager= .containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:319) =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.yarn.server= .nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch= .java:81)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.util.concur= rent.FutureTask$Sync.innerRun(FutureTask.java:314)
=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 at java.util.concurrent.FutureTask.run(FutureTask.jav= a:149)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.util.concurrent.ThreadPo= olExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.util.concurrent.ThreadPoolExecutor$Work= er.run(ThreadPoolExecutor.java:919)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 at java.lang.Thread.run(Thread.java:738)

Thanks!


2014-08-12 2:18 GMT-07:00 sam liu <samliuhadoop@gmail.c= om>:
Hi= Experts,

I am very interesting that Hadoop could work with Do= cker and doing some trial on patch of YARN-1964.

I applied patch yarn-1964-branch-2.2.0-docker.patch of jira = YARN-1964 on branch 2.2 and am going to install a Hadoop cluster using the = new generated tarball including the patch.

Then, I think I can use DockerContainerExecutor, but I do not know much= details on the usage and have following questions:

1. Afte= r installation, What's the detailed config steps to adopt DockerContain= erExecutor?

2. How to verify whether a MR task is really launched in D= ocker container not Yarn container?

3. Which hadoop bra= nch will officially include Docker support?

Thanks a lot!<= br>

--001a11c257aa83f4cb05007a2d84--