apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ananth Gundabattula <agundabatt...@gmail.com>
Subject Re: YARN memory settings and the Apex memory model
Date Thu, 12 May 2016 20:39:15 GMT
Thanks a lot Thomas & Sandesh.


I somehow missed the "troubleshooting" section of the docs. This is what I
was looking for.

Regards,
Ananth

On Fri, May 13, 2016 at 1:30 AM, Thomas Weise <thomas.weise@gmail.com>
wrote:

> Ananth,
>
> Please have a look at:
>
> http://docs.datatorrent.com/troubleshooting/#configuring-memory
>
> Thanks,
> Thomas
>
>
> On Thu, May 12, 2016 at 4:00 AM, Ananth Gundabattula <
> agundabattula@gmail.com> wrote:
>
>> Thanks Shubham. I shall bump up the memory a bit more.
>>
>> I was wondering how the operator memory relates to the YARN container
>> memory settings ? Or it depends on the deployment models ?
>>
>> For example , if the deployment model is thread local, the YARN container
>> needs to be ( considering above example ) configured for atleast memory of
>> 2048 * number of operators + Buffer Server Size ?
>>
>> If the deployment model were not Thread local, it would make YARN
>> container requirements for memory lower per container ?
>>
>> Regards,
>> Ananth
>>
>> On Thu, May 12, 2016 at 7:19 PM, Shubham Pathak <shubham@datatorrent.com>
>> wrote:
>>
>>> Hello Ananth,
>>>
>>> Looks like operator requires more memory.
>>> You may add this property to have more memory allocated to the container.
>>>
>>> In properties.xml , for operator O in the application you may specify
>>> the property :
>>>
>>> <property>
>>>  <name>dt.operator.*O*.attr.MEMORY_MB</name>
>>> <value>2048</value>
>>>  </property>
>>>
>>> Thanks,
>>> Shubham
>>>
>>> On Thu, May 12, 2016 at 1:35 PM, Ananth Gundabattula <
>>> agundabattula@gmail.com> wrote:
>>>
>>>> Hello All,
>>>>
>>>> I am seeing the following log from the web ui ocassionally when my
>>>> operators are getting killed. Is there any way  I can control the memory
>>>> settings that are used to communicate with YARN when negotiating a
>>>> container ?
>>>>
>>>> How does the typical yarn settings for a container heap and max memory
>>>> relate to the Apex memory allocation model.
>>>>
>>>> The info messages I see in the web UI are as follows:
>>>>
>>>> Container [pid=14699,containerID=container_1462863487071_0015_01_000012]
is running beyond physical memory limits. Current usage: 1.5 GB of 1.5 GB physical memory
used; 6.1 GB of 3.1 GB virtual memory used. Killing container.
>>>> Dump of the process-tree for container_1462863487071_0015_01_000012 :
>>>> 	|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS)
VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>>>> 	|- 14817 14699 14699 14699 (java) 1584 1654 6426968064 393896 /usr/java/default/bin/java
-Xmx4429185024 -Ddt.attr.APPLICATION_PATH=hdfs://dwh109.qaperf2.sac.int.threatmetrix.com:8020/user/dtadmin/datatorrent/apps/application_1462863487071_0015
-Djava.io.tmpdir=/data3/yarn/nm/usercache/root/appcache/application_1462863487071_0015/container_1462863487071_0015_01_000012/tmp
-Ddt.cid=container_1462863487071_0015_01_000012 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012
-Ddt.loggers.level=com.datatorrent.*:INFO,org.apache.*:INFO com.datatorrent.stram.engine.StreamingContainer
>>>> 	|- 14699 14697 14699 14699 (bash) 1 2 108646400 303 /bin/bash -c /usr/java/default/bin/java
 -Xmx4429185024  -Ddt.attr.APPLICATION_PATH=hdfs://dwh109.qaperf2.sac.int.threatmetrix.com:8020/user/dtadmin/datatorrent/apps/application_1462863487071_0015
-Djava.io.tmpdir=/data3/yarn/nm/usercache/root/appcache/application_1462863487071_0015/container_1462863487071_0015_01_000012/tmp
-Ddt.cid=container_1462863487071_0015_01_000012 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012
-Ddt.loggers.level=com.datatorrent.*:INFO,org.apache.*:INFO com.datatorrent.stram.engine.StreamingContainer
1>/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012/stdout
2>/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012/stderr
>>>>
>>>> Container killed on request. Exit code is 143
>>>> Container exited with a non-zero exit code 143
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Ananth
>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message