Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id A7F14200B49 for ; Tue, 19 Jul 2016 20:27:33 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id A68FD160A5C; Tue, 19 Jul 2016 18:27:33 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id C7D9C160A76 for ; Tue, 19 Jul 2016 20:27:32 +0200 (CEST) Received: (qmail 21040 invoked by uid 500); 19 Jul 2016 18:27:31 -0000 Mailing-List: contact users-help@apex.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@apex.apache.org Delivered-To: mailing list users@apex.apache.org Received: (qmail 20962 invoked by uid 99); 19 Jul 2016 18:27:31 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Jul 2016 18:27:31 +0000 Received: from mail-it0-f50.google.com (mail-it0-f50.google.com [209.85.214.50]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 723791A0055 for ; Tue, 19 Jul 2016 18:27:31 +0000 (UTC) Received: by mail-it0-f50.google.com with SMTP id f6so27886702ith.0 for ; Tue, 19 Jul 2016 11:27:31 -0700 (PDT) X-Gm-Message-State: ALyK8tKRA1qe+/IbcFYIn8x9DLvrXUdJJJQ6/OvDYm9C5gwBXjgB1UrEf3Ysywly8GHDZXsn4IvJLxivLsBtEWPc X-Received: by 10.36.110.212 with SMTP id w203mr37868993itc.62.1468952850387; Tue, 19 Jul 2016 11:27:30 -0700 (PDT) MIME-Version: 1.0 Received: by 10.79.38.15 with HTTP; Tue, 19 Jul 2016 11:27:29 -0700 (PDT) In-Reply-To: References: From: "Pradeep A. Dalvi" Date: Tue, 19 Jul 2016 11:27:29 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Container & memory resource allocation To: dev@apex.apache.org Cc: users@apex.apache.org Content-Type: multipart/alternative; boundary=001a114ac9c4d720d20538013ca3 archived-at: Tue, 19 Jul 2016 18:27:33 -0000 --001a114ac9c4d720d20538013ca3 Content-Type: text/plain; charset=UTF-8 Thanks Chinmay & Ram. Troubleshooting page sounds the appropriate location. I shall raise PR with the given suggestions. --prad On Tue, Jul 19, 2016 at 5:49 AM, Munagala Ramanath wrote: > There is already a link to a troubleshooting page at bottom of > https://apex.apache.org/docs.html > That page already has some discussion under the section entitled > "Calculating Container Memory" > so adding new content there seems like the right thing to do. > > Ram > > On Mon, Jul 18, 2016 at 11:27 PM, Chinmay Kolhatkar < > chinmay@datatorrent.com > > wrote: > > > Hi Pradeep, > > > > This is a great content to add to the documents. These are the common set > > of errors which might get googled and hence great to get indexed as well. > > > > You can take a look at: > > https://github.com/apache/apex-core/tree/master/docs > > > > The docs for apex reside there in markdown format. Probably its good a > > create a troubleshooting page where all such common questions can reside. > > > > After you have the content ready, you can create a pull request to > > apex-core repo which can get merged to apex-core and later deployed to > the > > website by committers. > > > > -Chinmay. > > > > > > > > > > On Tue, Jul 19, 2016 at 10:46 AM, Pradeep A. Dalvi > > wrote: > > > >> Container & memory resource allocation has been a common question around > >> and so I thought it would be good to explain related configuration > >> parameters. > >> > >> Please feel free to let me know your thoughts. > >> > >> Also I'm planning to add following set of information under Apex Docs. > How > >> could one add this to Apex Docs? > >> > >> =-=-=-= > >> > >> "Container is running beyond physical memory limits. Current usage: X GB > >> of > >> Y GB physical memory used; A GB of B GB virtual memory used. Killing > >> container." > >> > >> This is basically for some better understanding on Application Master's > >> container requests & Resource Manager's memory resource allocation. > Please > >> note that these are individual container request params. All these > >> parameters are in MB i.e. 1024 => 1GB. > >> > >> - AM's container requests to RM shall contain memory in the multiples of > >> *yarn.scheduler.minimum-**allocation-mb* & not exceeding > >> *yarn.scheduler.maximum-**allocation-mb* > >> - If *yarn.scheduler.minimum-**allocation-mb *is configured as 1024 > and > >> container memory requirement is 1025 ( <= 2048 ), container will be > >> allocated with 2048 memory. > >> > >> - With Apex applications, operator memory can be specified by property > >> *dt.application..operator..attr.MEMORY_MB* > >> - Please note this parameter is at Operator level and container > memory > >> is calculated based on number of Operators deployed in a container + > >> additional memory required depending on physical deployment requirements > >> e.g. unifier or bufferserver > >> - Wildcard * can be used at APP_NAME and/or OPERATOR_NAME > >> > >> - If container memory is not specified, then AM would request for 1 unit > >> of > >> *yarn.scheduler.minimum-**allocation-mb*, RM would provision container > >> taking that into consideration. > >> > >> Node Manager monitors memory usage of each of these containers and kills > >> the ones crossing the configured limit. > >> > >> Almost similar stuff is applicable for CPUs. > >> > >> --prad > >> > > > > > --001a114ac9c4d720d20538013ca3 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thanks Chinmay & Ram.

Troubleshooti= ng page sounds the appropriate location. I shall raise PR with the given su= ggestions.

--prad

On Tue, Jul 19, 2016 at 5:49 AM, Munag= ala Ramanath <ram@datatorrent.com> wrote:
There is already a link to a troubleshooting page at bott= om of
https://apex.apache.org/docs.html
That page already has some discussion under the section entitled
"Calculating Container Memory"
so adding new content there seems like the right thing to do.

Ram

On Mon, Jul 18, 2016 at 11:27 PM, Chinmay Kolhatkar <chinmay@datatorrent.com
> wrote:

> Hi Pradeep,
>
> This is a great content to add to the documents. These are the common = set
> of errors which might get googled and hence great to get indexed as we= ll.
>
> You can take a look at:
> https://github.com/apache/apex-core/tree/= master/docs
>
> The docs for apex reside there in markdown format. Probably its good a=
> create a troubleshooting page where all such common questions can resi= de.
>
> After you have the content ready, you can create a pull request to
> apex-core repo which can get merged to apex-core and later deployed to= the
> website by committers.
>
> -Chinmay.
>
>
>
>
> On Tue, Jul 19, 2016 at 10:46 AM, Pradeep A. Dalvi <prad@apache.org>
> wrote:
>
>> Container & memory resource allocation has been a common quest= ion around
>> and so I thought it would be good to explain related configuration=
>> parameters.
>>
>> Please feel free to let me know your thoughts.
>>
>> Also I'm planning to add following set of information under Ap= ex Docs. How
>> could one add this to Apex Docs?
>>
>> =3D-=3D-=3D-=3D
>>
>> "Container is running beyond physical memory limits. Current = usage: X GB
>> of
>> Y GB physical memory used; A GB of B GB virtual memory used. Killi= ng
>> container."
>>
>> This is basically for some better understanding on Application Mas= ter's
>> container requests & Resource Manager's memory resource al= location. Please
>> note that these are individual container request params. All these=
>> parameters are in MB i.e. 1024 =3D> 1GB.
>>
>> - AM's container requests to RM shall contain memory in the mu= ltiples of
>> *yarn.scheduler.minimum-**allocation-mb* & not exceeding
>> *yarn.scheduler.maximum-**allocation-mb*
>>=C2=A0 =C2=A0 - If *yarn.scheduler.minimum-**allocation-mb *is conf= igured as 1024 and
>> container memory requirement is 1025 ( <=3D 2048 ), container w= ill be
>> allocated with 2048 memory.
>>
>> - With Apex applications, operator memory can be specified by prop= erty
>> *dt.application.<APP_NAME>.operator.<OPERATOR_NAME>.at= tr.MEMORY_MB*
>>=C2=A0 =C2=A0 - Please note this parameter is at Operator level and= container memory
>> is calculated based on number of Operators deployed in a container= +
>> additional memory required depending on physical deployment requir= ements
>> e.g. unifier or bufferserver
>>=C2=A0 =C2=A0 - Wildcard * can be used at APP_NAME and/or OPERATOR_= NAME
>>
>> - If container memory is not specified, then AM would request for = 1 unit
>> of
>> *yarn.scheduler.minimum-**allocation-mb*, RM would provision conta= iner
>> taking that into consideration.
>>
>> Node Manager monitors memory usage of each of these containers and= kills
>> the ones crossing the configured limit.
>>
>> Almost similar stuff is applicable for CPUs.
>>
>> --prad
>>
>
>

--001a114ac9c4d720d20538013ca3--