Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DAF0C1093E for ; Tue, 2 Jul 2013 20:05:13 +0000 (UTC) Received: (qmail 19758 invoked by uid 500); 2 Jul 2013 20:05:09 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 19672 invoked by uid 500); 2 Jul 2013 20:05:08 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 19665 invoked by uid 99); 2 Jul 2013 20:05:08 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Jul 2013 20:05:08 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of sandy.ryza@cloudera.com designates 209.85.160.42 as permitted sender) Received: from [209.85.160.42] (HELO mail-pb0-f42.google.com) (209.85.160.42) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Jul 2013 20:05:04 +0000 Received: by mail-pb0-f42.google.com with SMTP id un1so6580041pbc.1 for ; Tue, 02 Jul 2013 13:04:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:x-gm-message-state; bh=ZX0OGTeu5auB6zIXKS9TSYnRbtNHOhECDga+S0txxcQ=; b=ax+qF38CHh/4UwhDIgVnWERS82+feveLJtZ9GmCCn09Jc9rk+s5H/Gt6uYdOZU+RFB XGBZsm6AK/TEJrBEVT6Aj0XxLvfP1qlnd3s/dp7WrqoKE9J+0aWBJ5uzZK71ZRHuwV7D fB5nmvr9hRriOgjj///n82fpMEzm2NRr5qZeDbHvSPT3GJ79CBLXybKPKD6WlAxpQR2t oiVNnklZb+FfCI35mAVPo95tWbvXb4FBAKjoXTkhkRRF5EpFp3nfcyc81I8TeqnYffTn wwBpS9bsIeBxbk2h7uotszAkl+xvCWPE12JRLeb1czuR7/ika5wuFY3Rf+vbBUKoR0+6 saHA== MIME-Version: 1.0 X-Received: by 10.66.233.168 with SMTP id tx8mr5588517pac.80.1372795484350; Tue, 02 Jul 2013 13:04:44 -0700 (PDT) Received: by 10.70.8.37 with HTTP; Tue, 2 Jul 2013 13:04:44 -0700 (PDT) In-Reply-To: <869970D71E26D7498BDAC4E1CA92226B658D6ABC@MBX021-E3-NJ-2.exch021.domain.local> References: <869970D71E26D7498BDAC4E1CA92226B658D64E9@MBX021-E3-NJ-2.exch021.domain.local> <869970D71E26D7498BDAC4E1CA92226B658D6A5F@MBX021-E3-NJ-2.exch021.domain.local> <869970D71E26D7498BDAC4E1CA92226B658D6ABC@MBX021-E3-NJ-2.exch021.domain.local> Date: Tue, 2 Jul 2013 13:04:44 -0700 Message-ID: Subject: Re: Containers and CPU From: Sandy Ryza To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=047d7b111cb531c08a04e08cdb89 X-Gm-Message-State: ALoCoQlzISqaOENzPRc3wbNDyJs9AQsh2BDR1dB2glRto9GAtdl0JOzaOQ7l155hv3LG0/TOVerF X-Virus-Checked: Checked by ClamAV on apache.org --047d7b111cb531c08a04e08cdb89 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable That's correct. -Sandy On Tue, Jul 2, 2013 at 12:28 PM, John Lilley wrot= e: > Sandy,**** > > Thanks, I think I understand. So it only makes a difference if cgroups i= s > on AND the AM requests multiple cores? E.g. if each task wants 4 cores t= he > RM would only allow two containers per 8-core node?**** > > John**** > > ** ** > > ** ** > > *From:* Sandy Ryza [mailto:sandy.ryza@cloudera.com] > *Sent:* Tuesday, July 02, 2013 1:26 PM > > *To:* user@hadoop.apache.org > *Subject:* Re: Containers and CPU**** > > ** ** > > Use of cgroups for controlling CPU is off by default, but can be turned o= n > as a nodemanager configuration with > yarn.nodemanager.linux-container-executor.resources-handler.class. So it > is site-wide. If you want tasks to purely fight it out in the OS thread > scheduler, simply don't change from the default.**** > > ** ** > > Even with cgroups on, all tasks will have access to all CPU cores. We > don't do any pinning of tasks to cores. If a task is requested with a > single vcore and placed on an otherwise empty machine with 8 cores, it wi= ll > have access to all 8 cores. If 3 other tasks that requested a single vco= re > are later placed on the same node, and all tasks are using as much CPU as > they can get their hands on, then each of the tasks will get 2 cores of > CPU-time.**** > > ** ** > > On Tue, Jul 2, 2013 at 12:12 PM, John Lilley > wrote:**** > > Sandy,**** > > Sorry, I don=92t completely follow. **** > > When you say =93with cgroups on=94, is that an attribute of the AM, the > Scheduler, or the Site/RM? In other words is it site-wide or something > that my application can control?**** > > With cgroups on, is there still a way to get my desired behavior? I=92d > really like all tasks to have access to all CPU cores and simply fight it > out in the OS thread scheduler.**** > > Thanks,**** > > john**** > > **** > > *From:* Sandy Ryza [mailto:sandy.ryza@cloudera.com] > *Sent:* Tuesday, July 02, 2013 11:56 AM > *To:* user@hadoop.apache.org > *Subject:* Re: Containers and CPU**** > > **** > > CPU limits are only enforced if cgroups is turned on. With cgroups on, > they are only limited when there is contention, in which case tasks are > given CPU time in proportion to the number of cores requested for/allocat= ed > to them. Does that make sense?**** > > **** > > -Sandy**** > > **** > > On Tue, Jul 2, 2013 at 9:50 AM, Chuan Liu wrote:= * > *** > > I believe this is the default behavior.**** > > By default, only memory limit on resources is enforced.**** > > The capacity scheduler will use DefaultResourceCalculator to compute > resource allocation for containers by default, which also does not take C= PU > into account.**** > > **** > > -Chuan**** > > **** > > *From:* John Lilley [mailto:john.lilley@redpoint.net] > *Sent:* Tuesday, July 02, 2013 8:57 AM > *To:* user@hadoop.apache.org > *Subject:* Containers and CPU**** > > **** > > I have YARN tasks that benefit from multicore scaling. However, they > don=92t **always** use more than one core. I would like to allocate > containers based only on memory, and let each task use as many cores as > needed, without allocating exclusive CPU =93slots=94 in the scheduler. F= or > example, on an 8-core node with 16GB memory, I=92d like to be able to run= 3 > tasks each consuming 4GB memory and each using as much CPU as they like. > Is this the default behavior if I don=92t specify CPU restrictions to the > scheduler?**** > > Thanks**** > > John**** > > **** > > **** > > **** > > ** ** > --047d7b111cb531c08a04e08cdb89 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable
That's correct.

-Sandy
<= /div>


On Tue, = Jul 2, 2013 at 12:28 PM, John Lilley <john.lilley@redpoint.net&= gt; wrote:

Sandy,

Thanks, I think I underst= and.=A0 So it only makes a difference if cgroups is on AND the AM requests = multiple cores?=A0 E.g. if each task wants 4 cores the RM would only allow two containers per 8-core node?

John=

=A0<= /p>

=A0<= /p>

From: Sandy Ry= za [mailto:san= dy.ryza@cloudera.com]
Sent: Tuesday, July 02, 2013 1:26 PM


To: user= @hadoop.apache.org
Subject: Re: Containers and CPU

=A0

Use of cgroups for controlling CPU is off by default= , but can be turned on as a nodemanager configuration with yarn.nodemanager= .linux-container-executor.resources-handler.class. =A0So it is site-wide. = =A0If you want tasks to purely fight it out in the OS thread scheduler, simply don't change from the default.<= u>

=A0

Even with cgroups on, all tasks will have access to = all CPU cores. =A0We don't do any pinning of tasks to cores. =A0If a ta= sk is requested with a single vcore and placed on an otherwise empty machin= e with 8 cores, it will have access to all 8 cores. =A0If 3 other tasks that requested a single vcore are later place= d on the same node, and all tasks are using as much CPU as they can get the= ir hands on, then each of the tasks will get 2 cores of CPU-time.=

=A0

On Tue, Jul 2, 2013 at 12:12 PM, John Lilley <john.lilley@redp= oint.net> wrote:

Sandy,

Sorry, I don=92t complete= ly follow.=A0

When you say =93with cgro= ups on=94, is that an attribute of the AM, the Scheduler, or the Site/RM?= =A0 In other words is it site-wide or something that my application can control?<= /span>

With cgroups on, is there= still a way to get my desired behavior?=A0 I=92d really like all tasks to = have access to all CPU cores and simply fight it out in the OS thread scheduler= .

Thanks,<= /u>

john=

=A0<= /p>

From: Sandy Ry= za [mailto:san= dy.ryza@cloudera.com]
Sent: Tuesday, July 02, 2013 11:56 AM
To: user= @hadoop.apache.org
Subject: Re: Containers and CPU

=A0

CPU limits are only enforced if cgroups is turned on= . =A0With cgroups on, they are only limited when there is contention, in wh= ich case tasks are given CPU time in proportion to the number of cores requested for/allocated to them. =A0Does that make sense?<= u>

=A0

-Sandy

=A0

On Tue, Jul 2, 2013 at 9:50 AM, Chuan Liu <chuanliu@microsoft.co= m> wrote:

I believe this is the = default behavior.

By default, only memor= y limit on resources is enforced.

The capacity scheduler= will use DefaultResourceCalculator to compute resource allocation for cont= ainers by default, which also does not take CPU into account.

=A0

-Chuan

=A0

From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Tuesday, July 02, 2013 8:57 AM
To: user= @hadoop.apache.org
Subject: Containers and CPU

=A0

I have YARN tasks that benefit from multicore scalin= g.=A0 However, they don=92t *always* use more than one core.=A0 I wo= uld like to allocate containers based only on memory, and let each task use as many cores as needed, without allocating exclusive CP= U =93slots=94 in the scheduler.=A0 For example, on an 8-core node with 16GB= memory, I=92d like to be able to run 3 tasks each consuming 4GB memory and= each using as much CPU as they like.=A0 Is this the default behavior if I don=92t specify CPU restrictions to the sch= eduler?

Thanks

John

=A0

=A0

=A0

=A0


--047d7b111cb531c08a04e08cdb89--