apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chinmay Kolhatkar <chin...@datatorrent.com>
Subject Re: Container & memory resource allocation
Date Tue, 19 Jul 2016 06:27:11 GMT
Hi Pradeep,

This is a great content to add to the documents. These are the common set
of errors which might get googled and hence great to get indexed as well.

You can take a look at: https://github.com/apache/apex-core/tree/master/docs

The docs for apex reside there in markdown format. Probably its good a
create a troubleshooting page where all such common questions can reside.

After you have the content ready, you can create a pull request to
apex-core repo which can get merged to apex-core and later deployed to the
website by committers.

-Chinmay.




On Tue, Jul 19, 2016 at 10:46 AM, Pradeep A. Dalvi <prad@apache.org> wrote:

> Container & memory resource allocation has been a common question around
> and so I thought it would be good to explain related configuration
> parameters.
>
> Please feel free to let me know your thoughts.
>
> Also I'm planning to add following set of information under Apex Docs. How
> could one add this to Apex Docs?
>
> =-=-=-=
>
> "Container is running beyond physical memory limits. Current usage: X GB of
> Y GB physical memory used; A GB of B GB virtual memory used. Killing
> container."
>
> This is basically for some better understanding on Application Master's
> container requests & Resource Manager's memory resource allocation. Please
> note that these are individual container request params. All these
> parameters are in MB i.e. 1024 => 1GB.
>
> - AM's container requests to RM shall contain memory in the multiples of
> *yarn.scheduler.minimum-**allocation-mb* & not exceeding
> *yarn.scheduler.maximum-**allocation-mb*
>    - If *yarn.scheduler.minimum-**allocation-mb *is configured as 1024 and
> container memory requirement is 1025 ( <= 2048 ), container will be
> allocated with 2048 memory.
>
> - With Apex applications, operator memory can be specified by property
> *dt.application.<APP_NAME>.operator.<OPERATOR_NAME>.attr.MEMORY_MB*
>    - Please note this parameter is at Operator level and container memory
> is calculated based on number of Operators deployed in a container +
> additional memory required depending on physical deployment requirements
> e.g. unifier or bufferserver
>    - Wildcard * can be used at APP_NAME and/or OPERATOR_NAME
>
> - If container memory is not specified, then AM would request for 1 unit of
> *yarn.scheduler.minimum-**allocation-mb*, RM would provision container
> taking that into consideration.
>
> Node Manager monitors memory usage of each of these containers and kills
> the ones crossing the configured limit.
>
> Almost similar stuff is applicable for CPUs.
>
> --prad
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message