Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A43501864C for ; Sun, 23 Aug 2015 20:18:29 +0000 (UTC) Received: (qmail 51124 invoked by uid 500); 23 Aug 2015 20:18:23 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 51003 invoked by uid 500); 23 Aug 2015 20:18:23 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 50993 invoked by uid 99); 23 Aug 2015 20:18:23 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 23 Aug 2015 20:18:23 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id BD053182344 for ; Sun, 23 Aug 2015 20:18:22 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.898 X-Spam-Level: ** X-Spam-Status: No, score=2.898 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id aCYUTuz57hz8 for ; Sun, 23 Aug 2015 20:18:20 +0000 (UTC) Received: from mail-oi0-f47.google.com (mail-oi0-f47.google.com [209.85.218.47]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id 5A74F20752 for ; Sun, 23 Aug 2015 20:18:20 +0000 (UTC) Received: by oiew67 with SMTP id w67so69120958oie.2 for ; Sun, 23 Aug 2015 13:18:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=8M8AwqLIBWaL7IQUrB6o0emxaz2wyyl6aLlcU8snCvU=; b=P4WJWcW5dU/XLrX7ksGOmIckBwQvyeJy1xLSujyxelNxk5sqkVHF4greHa6ekagFk3 9rpshdmAdhdAdNIJRKz2h1PaK/mYB+CzBl6NDLT6IgJx8l4ZmM0XKOyND0Q1QYReEbsm WsWC1zq4LJ8vRKQJrHTgLFOZQMjHO+TDIorJ0sfbAeMAk7Ro2Me43fW7ri4knd0KYTN0 IL+j4/FzUk0pbWfgyOMKJxngQKKZKRa3yUOdvE2f1rD9XRIoD3/Z0GEd2BW2+sH6gCRJ Xmc3X44Q3Qh7iWXe2RLmtjVSkodnBXUeU4im39doKmKgCCrXbxujMgu34tioo0bhqvTb XADw== MIME-Version: 1.0 X-Received: by 10.202.214.72 with SMTP id n69mr17119787oig.67.1440361099652; Sun, 23 Aug 2015 13:18:19 -0700 (PDT) Received: by 10.202.177.70 with HTTP; Sun, 23 Aug 2015 13:18:19 -0700 (PDT) In-Reply-To: References: Date: Sun, 23 Aug 2015 17:18:19 -0300 Message-ID: Subject: Re: yarn.nodemanager.resource.cpu-vcores vs yarn.scheduler.maximum-allocation-vcores From: Pedro Magalhaes To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a113b1c02b17b0a051e003315 --001a113b1c02b17b0a051e003315 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable So if I have nodes with different yarn.nodemanager.resource.cpu-vcores and yarn.nodemanager.resource.memory-mb, the configuration will show me all different values? Or will just show me just the default values? On Sun, Aug 23, 2015 at 4:56 PM, Varun Saxena wrote: > This configuration is read and used by NodeManager, on whichever node its > running. > If it is not configured, default value will be taken. > > Regards, > Varun Saxena. > > On Mon, Aug 24, 2015 at 1:21 AM, Pedro Magalhaes > wrote: > >> Thanks Varun! Like we say in Brazil. "U are the guy!" (Voc=C3=AA =C3=A9= o cara!) >> >> I have another question. You said that: >> "yarn.nodemanager.resource.cpu-vcores on the other hand will have to be >> configured as per resource capability of that particular node. " >> >> I get the configuration from my job and printed it: >> yarn.nodemanager.resource.cpu-vcores 8 >> yarn.nodemanager.resource.memory-mb 8192 >> >> So how does hadoop get this property if it is per node? Does it get the >> minimum of all nodes? Thanks again! >> >> >> >> On Sun, Aug 23, 2015 at 4:40 PM, Varun Saxena >> wrote: >> >>> The fix would be released in next version(2.8.0). >>> I had checked the code to find out the default value and then found it >>> fixed in documentation(configuration list). >>> >>> As this is an unreleased version, a URL link (of the form >>> https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-common/ya= rn-default.xml) >>> may not be available AFAIK, >>> However, this XML(yarn-default.xml) can be checked online in git >>> repository. >>> >>> Associated JIRA which fixes this is >>> https://issues.apache.org/jira/browse/YARN-3823 >>> >>> Regards, >>> Varun Saxena. >>> >>> On Mon, Aug 24, 2015 at 12:53 AM, Pedro Magalhaes >>> wrote: >>> >>>> Thanks Varun! >>>> Could plz send me the link with the fixed? >>>> >>>> On Sun, Aug 23, 2015 at 2:20 PM, Varun Saxena >>>> wrote: >>>> >>>>> Hi Pedro, >>>>> >>>>> Real default value of yarn.scheduler.maximum-allocation-vcores is 4. >>>>> The value of 32 is actually a documentation issue and has been fixed >>>>> recently. >>>>> >>>>> Regards, >>>>> Varun Saxena. >>>>> >>>>> >>>>> On Sun, Aug 23, 2015 at 10:39 PM, Pedro Magalhaes >>>> > wrote: >>>>> >>>>>> Varun, >>>>>> Thanks for the reply. I undestand the arn.scheduler.maximum- >>>>>> allocation-vcores parameter. I just asking why the default parameter >>>>>> is yarn.scheduler.maximum-allocation-vcores=3D32. And >>>>>> yarn.nodemanager.resource.cpu-vcores=3D8. >>>>>> >>>>>> In my opinion, if the yarn.scheduler.maximun-allocation-vcore is 32 >>>>>> tby default the yarn.nodemanager.resource.cpu-vcores would be equal= or >>>>>> greater than 32, by default. >>>>>> Is this make sense? >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Sun, Aug 23, 2015 at 2:00 PM, Varun Saxena < >>>>>> vsaxena.varun@gmail.com> wrote: >>>>>> >>>>>>> Hi Pedro, >>>>>>> >>>>>>> Actual allocation would depend on the total resource capability >>>>>>> advertised by NM while registering with RM. >>>>>>> >>>>>>> yarn.scheduler.maximum-allocation-vcores merely puts an upper cap o= n number of vcores which can be allocated by RM i.e. any Resource request/a= sk from AM which asks for vcores > 32(default value) for a container, will = be normalized back to 32. >>>>>>> >>>>>>> If there is no such node available, this allocation will not be ful= filled. >>>>>>> >>>>>>> yarn.scheduler.maximum-allocation-vcores will be configured in >>>>>>> resource manager and hence will be common for a cluster which can p= ossibly >>>>>>> have multiple nodes with heterogeneous resource capabilities >>>>>>> >>>>>>> yarn.nodemanager.resource.cpu-vcores on the other hand will have to >>>>>>> be configured as per resource capability of that particular node. >>>>>>> >>>>>>> Recently there has been work done to automatically get memory and >>>>>>> CPU information from underlying OS(supported OS being Linux and Win= dows) if >>>>>>> configured to do so. This change would be available in 2.8 >>>>>>> I hope this answers your question. >>>>>>> >>>>>>> Regards, >>>>>>> Varun Saxena. >>>>>>> >>>>>>> On Sun, Aug 23, 2015 at 9:40 PM, Pedro Magalhaes < >>>>>>> pedrorjbr@gmail.com> wrote: >>>>>>> >>>>>>>> I was looking at default parameters for: >>>>>>>> >>>>>>>> yarn.nodemanager.resource.cpu-vcores =3D 8 >>>>>>>> yarn.scheduler.maximum-allocation-vcores =3D 32 >>>>>>>> >>>>>>>> For me this two parameters as default doesnt make any sense. >>>>>>>> >>>>>>>> The first one say "the number of CPU cores that can be allocated >>>>>>>> for containers." (I imagine that is vcore) The seconds says: "The = maximum >>>>>>>> allocation for every container request at the RM". In my opinion, = the >>>>>>>> second one must be equal or less than the first one. >>>>>>>> >>>>>>>> How can allocate 32 vcores for a container if i have only 8 cores >>>>>>>> available per container? >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > --001a113b1c02b17b0a051e003315 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
So if I have nodes with different yarn.nodemanager.resourc= e.cpu-vcores and yarn.nodemanager.resource.memory-mb, the configuration wil= l show me all different values? Or will just show me just the default value= s?




On Sun, Aug 23, 2015 at 4:5= 6 PM, Varun Saxena <vsaxena.varun@gmail.com> wrote:
This configuration is read= and used by NodeManager, on whichever node its running.
If it is not c= onfigured, default value will be taken.

Regards,
Varun Saxena.
<= div class=3D"gmail_extra">
On Mon, Aug 24, 20= 15 at 1:21 AM, Pedro Magalhaes <pedrorjbr@gmail.com> wrote= :
Thanks Varun! Like we = say in Brazil. =C2=A0"U are the guy!" (Voc=C3=AA =C3=A9 o cara!)<= br>
I have another question. You said that:
"y= arn.nodemanager.resource.cpu-vcores on the other hand will have to be configured as per resource c= apability of that particular node.=C2=A0"

I get the configuration from my job and printed i= t:
yarn.nodemanager.r= esource.cpu-vcores 8=20 yarn.nodemanager.resource.memory-mb 8192

So how does hadoop get this property if it is per node? Does it g= et the minimum of all nodes? Thanks again!



On Sun, Aug 23, 2015 at 4:40 PM, Varun Saxena <vsaxena.varun@gmail.com> wrote:
The fix would be released in next version(2.8.0).= =C2=A0
I had checked the code to find out the default value and then fo= und it fixed in documentation(configuration list).=C2=A0

=
As this is an unreleased version, a URL link (of the form https://hadoop.apache.org/docs/r2.7.1/ha= doop-yarn/hadoop-yarn-common/yarn-default.xml) may not be available AFA= IK,
However, this XML(yarn-default.xml) can be checked online in = git repository.

Associated JIRA which fixes this i= s=C2=A0https://issues.apache.org/jira/browse/YARN-3823
<= br>
Regards,
Varun Saxena.

On Mon, Aug 24, 2015 at = 12:53 AM, Pedro Magalhaes <pedrorjbr@gmail.com> wrote:
=
Thanks Varun!
Could plz= send me the link with the fixed?

On Sun, Aug 23, 2015 at 2:20 PM, Varu= n Saxena <vsaxena.varun@gmail.com> wrote:
Hi Pedro,=

Real = default value of yarn.scheduler.maximum-allocation-vcores is 4.
The value of 32 is actually a documentation issue and = has been fixed recently.

Regards,
Varun Saxena.


On Sun, Aug 23, 2015 at 10:39 PM, Pedro M= agalhaes <pedrorjbr@gmail.com> wrote:
Varun,
Thanks for the reply. I undestand= the=C2=A0arn.scheduler.m= aximum-allocation-= vcores=C2=A0parameter. I just asking why the default parameter is yarn.scheduler.maximum-allocation-vcores=3D32= .=C2=A0And=C2=A0ya= rn.nodemanager.resource.cpu-vcores=3D8.

In my opinion, if the= yarn.scheduler.maximun-allocation-vcore is 32 tby default the yarn.nodeman= ager.resource.cpu-vcores =C2=A0would be equal or greater than 32, by defaul= t.
Is this make sense?



On Sun, Aug 23, 2015 at 2:00 PM, Varun Saxena <= vsaxena.varun@= gmail.com> wrote:
Hi Pedro,

Actual allocation would depend on the= total resource capability advertised by NM while registering with RM.
yarn.scheduler.maximum-allocation-vco=
res merely puts an upper cap on number of vcores which can be allocated by =
RM i.e. any Resource request/ask from AM which asks for vcores > 32(defa=
ult value) for a container, will be normalized back to 32.
If there is no such node available, this alloca= tion will not be fulfilled.
yarn.scheduler.maximum-a= llocation-vcores will be configured in resource manager and hence will be c= ommon for a cluster which can possibly have multiple nodes with heterogeneo= us resource capabilities

yarn.nodemanager.resource= .cpu-vcores on the other hand will have to be configured as per resource ca= pability of that particular node.=C2=A0

Recently t= here has been work done to automatically get memory and CPU information fro= m underlying OS(supported OS being Linux and Windows) if configured to do s= o. This change would be available in 2.8
I hope this answers your= question.

Regards,
Varun Saxena.
<= /div>

On= Sun, Aug 23, 2015 at 9:40 PM, Pedro Magalhaes <pedrorjbr@gmail.com&= gt; wrote:

I was looking at default parameters for:

yarn.nodeman=
ager.resource.cpu-vcores =3D 8
yarn.scheduler.maximum-allocation-vcores =3D 32

For me this two parameters as default doesnt mak= e any sense.

The first one say "the number of CPU cor= es that can be allocated for containers." (I imagine that is vcore) Th= e seconds says: "The maximum allocation for every container request at= the RM". In my opinion, the second one must be equal or less than the= first one.

How can allocate 32 vcores for a container if = i have only 8 cores available per container?









--001a113b1c02b17b0a051e003315--