apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ananth Gundabattula <agundabatt...@gmail.com>
Subject Re: YARN memory settings and the Apex memory model
Date Thu, 12 May 2016 11:00:12 GMT
Thanks Shubham. I shall bump up the memory a bit more.

I was wondering how the operator memory relates to the YARN container
memory settings ? Or it depends on the deployment models ?

For example , if the deployment model is thread local, the YARN container
needs to be ( considering above example ) configured for atleast memory of
2048 * number of operators + Buffer Server Size ?

If the deployment model were not Thread local, it would make YARN container
requirements for memory lower per container ?

Regards,
Ananth

On Thu, May 12, 2016 at 7:19 PM, Shubham Pathak <shubham@datatorrent.com>
wrote:

> Hello Ananth,
>
> Looks like operator requires more memory.
> You may add this property to have more memory allocated to the container.
>
> In properties.xml , for operator O in the application you may specify the
> property :
>
> <property>
>  <name>dt.operator.*O*.attr.MEMORY_MB</name>
> <value>2048</value>
>  </property>
>
> Thanks,
> Shubham
>
> On Thu, May 12, 2016 at 1:35 PM, Ananth Gundabattula <
> agundabattula@gmail.com> wrote:
>
>> Hello All,
>>
>> I am seeing the following log from the web ui ocassionally when my
>> operators are getting killed. Is there any way  I can control the memory
>> settings that are used to communicate with YARN when negotiating a
>> container ?
>>
>> How does the typical yarn settings for a container heap and max memory
>> relate to the Apex memory allocation model.
>>
>> The info messages I see in the web UI are as follows:
>>
>> Container [pid=14699,containerID=container_1462863487071_0015_01_000012] is running
beyond physical memory limits. Current usage: 1.5 GB of 1.5 GB physical memory used; 6.1 GB
of 3.1 GB virtual memory used. Killing container.
>> Dump of the process-tree for container_1462863487071_0015_01_000012 :
>> 	|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES)
RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>> 	|- 14817 14699 14699 14699 (java) 1584 1654 6426968064 393896 /usr/java/default/bin/java
-Xmx4429185024 -Ddt.attr.APPLICATION_PATH=hdfs://dwh109.qaperf2.sac.int.threatmetrix.com:8020/user/dtadmin/datatorrent/apps/application_1462863487071_0015
-Djava.io.tmpdir=/data3/yarn/nm/usercache/root/appcache/application_1462863487071_0015/container_1462863487071_0015_01_000012/tmp
-Ddt.cid=container_1462863487071_0015_01_000012 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012
-Ddt.loggers.level=com.datatorrent.*:INFO,org.apache.*:INFO com.datatorrent.stram.engine.StreamingContainer
>> 	|- 14699 14697 14699 14699 (bash) 1 2 108646400 303 /bin/bash -c /usr/java/default/bin/java
 -Xmx4429185024  -Ddt.attr.APPLICATION_PATH=hdfs://dwh109.qaperf2.sac.int.threatmetrix.com:8020/user/dtadmin/datatorrent/apps/application_1462863487071_0015
-Djava.io.tmpdir=/data3/yarn/nm/usercache/root/appcache/application_1462863487071_0015/container_1462863487071_0015_01_000012/tmp
-Ddt.cid=container_1462863487071_0015_01_000012 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012
-Ddt.loggers.level=com.datatorrent.*:INFO,org.apache.*:INFO com.datatorrent.stram.engine.StreamingContainer
1>/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012/stdout
2>/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012/stderr
>>
>> Container killed on request. Exit code is 143
>> Container exited with a non-zero exit code 143
>>
>>
>> Regards,
>>
>> Ananth
>>
>>
>>
>

Mime
View raw message