hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Naganarasimha G R (Naga)" <garlanaganarasi...@huawei.com>
Subject RE: YARN container killed as running beyond memory limits
Date Wed, 17 Jun 2015 12:23:22 GMT
Hi,
   From the logs its pretty clear its due to
"Current usage: 576.2 MB of 2 GB physical memory used; 4.2 GB of 4.2 GB virtual memory used.
Killing container."
Please increase the value yarn.nodemanager.vmem-pmem-ratio from the default value 2 to something
like 4 or 8 based on ur app and system.

+ Naga
________________________________
From: Arbi Akhina [arbi.akhina@gmail.com]
Sent: Wednesday, June 17, 2015 17:19
To: user@hadoop.apache.org
Subject: YARN container killed as running beyond memory limits

Hi, I've a YARN application that submits containers. In the AplicationMaster logs I see that
the container is killed. Here is the logs:


Jun 17, 2015 1:31:27 PM com.heavenize.modules.RMCallbackHandler onContainersCompleted
INFO: container 'container_1434471275225_0007_01_000002' status is ContainerStatus: [ContainerId:
container_1434471275225_0007_01_000002, State: COMPLETE, Diagnostics: Container [pid=4069,containerID=container_1434471275225_0007_01_000002]
is running beyond virtual memory limits. Current usage: 576.2 MB of 2 GB physical memory used;
4.2 GB of 4.2 GB virtual memory used. Killing container.
Dump of the process-tree for container_1434471275225_0007_01_000002 :
        |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES)
RSSMEM_USAGE(PAGES) FULL_CMD_LINE
        |- 4094 4093 4069 4069 (java) 2932 94 2916065280 122804 /usr/lib/jvm/java-7-openjdk-amd64/bin/java
-Xms512m -Xmx2048m -XX:MaxPermSize=250m -XX:+UseConcMarkSweepGC -Dosmoze.path=/tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/Osmoze
-Dspring.profiles.active=webServer -jar /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/heavenize-modules.jar
        |- 4093 4073 4069 4069 (sh) 0 0 4550656 164 /bin/sh /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/startup.sh
        |- 4073 4069 4069 4069 (java) 249 34 1577267200 24239 /usr/lib/jvm/java-7-openjdk-amd64/bin/java
com.heavenize.yarn.task.ModulesManager -containerId container_1434471275225_0007_01_000002
-port 5369 -exe hdfs://hadoop-server/user/hadoop/heavenize/heavenize-modules.jar -conf hdfs://hadoop-server/user/hadoop/heavenize/config.zip
        |- 4069 1884 4069 4069 (bash) 0 0 12730368 304 /bin/bash -c /usr/lib/jvm/java-7-openjdk-amd64/bin/java
com.heavenize.yarn.task.ModulesManager -containerId container_1434471275225_0007_01_000002
-port 5369 -exe hdfs://hadoop-server/user/hadoop/heavenize/heavenize-modules.jar -conf hdfs://hadoop-server/user/hadoop/heavenize/config.zip
1> /usr/local/hadoop/logs/userlogs/application_1434471275225_0007/container_1434471275225_0007_01_000002/stdout
2> /usr/local/hadoop/logs/userlogs/application_1434471275225_0007/container_1434471275225_0007_01_000002/stderr


I don't see any memory excess, any idea where this error comes from?
There is no errors in the container, it just stop logging as a result of being killed.

Mime
View raw message