apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pradeep A. Dalvi" <p...@apache.org>
Subject Re: Container & memory resource allocation
Date Tue, 19 Jul 2016 18:27:29 GMT
Thanks Chinmay & Ram.

Troubleshooting page sounds the appropriate location. I shall raise PR with
the given suggestions.

--prad

On Tue, Jul 19, 2016 at 5:49 AM, Munagala Ramanath <ram@datatorrent.com>
wrote:

> There is already a link to a troubleshooting page at bottom of
> https://apex.apache.org/docs.html
> That page already has some discussion under the section entitled
> "Calculating Container Memory"
> so adding new content there seems like the right thing to do.
>
> Ram
>
> On Mon, Jul 18, 2016 at 11:27 PM, Chinmay Kolhatkar <
> chinmay@datatorrent.com
> > wrote:
>
> > Hi Pradeep,
> >
> > This is a great content to add to the documents. These are the common set
> > of errors which might get googled and hence great to get indexed as well.
> >
> > You can take a look at:
> > https://github.com/apache/apex-core/tree/master/docs
> >
> > The docs for apex reside there in markdown format. Probably its good a
> > create a troubleshooting page where all such common questions can reside.
> >
> > After you have the content ready, you can create a pull request to
> > apex-core repo which can get merged to apex-core and later deployed to
> the
> > website by committers.
> >
> > -Chinmay.
> >
> >
> >
> >
> > On Tue, Jul 19, 2016 at 10:46 AM, Pradeep A. Dalvi <prad@apache.org>
> > wrote:
> >
> >> Container & memory resource allocation has been a common question around
> >> and so I thought it would be good to explain related configuration
> >> parameters.
> >>
> >> Please feel free to let me know your thoughts.
> >>
> >> Also I'm planning to add following set of information under Apex Docs.
> How
> >> could one add this to Apex Docs?
> >>
> >> =-=-=-=
> >>
> >> "Container is running beyond physical memory limits. Current usage: X GB
> >> of
> >> Y GB physical memory used; A GB of B GB virtual memory used. Killing
> >> container."
> >>
> >> This is basically for some better understanding on Application Master's
> >> container requests & Resource Manager's memory resource allocation.
> Please
> >> note that these are individual container request params. All these
> >> parameters are in MB i.e. 1024 => 1GB.
> >>
> >> - AM's container requests to RM shall contain memory in the multiples of
> >> *yarn.scheduler.minimum-**allocation-mb* & not exceeding
> >> *yarn.scheduler.maximum-**allocation-mb*
> >>    - If *yarn.scheduler.minimum-**allocation-mb *is configured as 1024
> and
> >> container memory requirement is 1025 ( <= 2048 ), container will be
> >> allocated with 2048 memory.
> >>
> >> - With Apex applications, operator memory can be specified by property
> >> *dt.application.<APP_NAME>.operator.<OPERATOR_NAME>.attr.MEMORY_MB*
> >>    - Please note this parameter is at Operator level and container
> memory
> >> is calculated based on number of Operators deployed in a container +
> >> additional memory required depending on physical deployment requirements
> >> e.g. unifier or bufferserver
> >>    - Wildcard * can be used at APP_NAME and/or OPERATOR_NAME
> >>
> >> - If container memory is not specified, then AM would request for 1 unit
> >> of
> >> *yarn.scheduler.minimum-**allocation-mb*, RM would provision container
> >> taking that into consideration.
> >>
> >> Node Manager monitors memory usage of each of these containers and kills
> >> the ones crossing the configured limit.
> >>
> >> Almost similar stuff is applicable for CPUs.
> >>
> >> --prad
> >>
> >
> >
>

Mime
View raw message