apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pradeep A. Dalvi" <p...@apache.org>
Subject Container & memory resource allocation
Date Tue, 19 Jul 2016 05:16:53 GMT
Container & memory resource allocation has been a common question around
and so I thought it would be good to explain related configuration
parameters.

Please feel free to let me know your thoughts.

Also I'm planning to add following set of information under Apex Docs. How
could one add this to Apex Docs?

=-=-=-=

"Container is running beyond physical memory limits. Current usage: X GB of
Y GB physical memory used; A GB of B GB virtual memory used. Killing
container."

This is basically for some better understanding on Application Master's
container requests & Resource Manager's memory resource allocation. Please
note that these are individual container request params. All these
parameters are in MB i.e. 1024 => 1GB.

- AM's container requests to RM shall contain memory in the multiples of
*yarn.scheduler.minimum-**allocation-mb* & not exceeding
*yarn.scheduler.maximum-**allocation-mb*
   - If *yarn.scheduler.minimum-**allocation-mb *is configured as 1024 and
container memory requirement is 1025 ( <= 2048 ), container will be
allocated with 2048 memory.

- With Apex applications, operator memory can be specified by property
*dt.application.<APP_NAME>.operator.<OPERATOR_NAME>.attr.MEMORY_MB*
   - Please note this parameter is at Operator level and container memory
is calculated based on number of Operators deployed in a container +
additional memory required depending on physical deployment requirements
e.g. unifier or bufferserver
   - Wildcard * can be used at APP_NAME and/or OPERATOR_NAME

- If container memory is not specified, then AM would request for 1 unit of
*yarn.scheduler.minimum-**allocation-mb*, RM would provision container
taking that into consideration.

Node Manager monitors memory usage of each of these containers and kills
the ones crossing the configured limit.

Almost similar stuff is applicable for CPUs.

--prad

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message