Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
MIME-Version: 1.0
Date: Wed, 17 Jun 2015 13:49:49 +0200
Message-ID: 
 <CAEBL=SBmYkD9pfk0w+fs8WwSUnTOGnsJu9T2C0Ep_-X6X0-Gcw@mail.gmail.com>
Subject: YARN container killed as running beyond memory limits
From: Arbi Akhina <arbi.akhina@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001a11475d4cc26aa70518b54932

--001a11475d4cc26aa70518b54932
Content-Type: text/plain; charset=UTF-8

Hi, I've a YARN application that submits containers. In the
AplicationMaster logs I see that the container is killed. Here is the logs:

Jun 17, 2015 1:31:27 PM com.heavenize.modules.RMCallbackHandler
onContainersCompleted
INFO: container 'container_1434471275225_0007_01_000002' status is
ContainerStatus: [ContainerId: container_1434471275225_0007_01_000002,
State: COMPLETE, Diagnostics: Container
[pid=4069,containerID=container_1434471275225_0007_01_000002] is
running beyond virtual memory limits. Current usage: 576.2 MB of 2 GB
physical memory used; 4.2 GB of 4.2 GB virtual memory used. Killing
container.
Dump of the process-tree for container_1434471275225_0007_01_000002 :
	|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES)
FULL_CMD_LINE
	|- 4094 4093 4069 4069 (java) 2932 94 2916065280 122804
/usr/lib/jvm/java-7-openjdk-amd64/bin/java -Xms512m -Xmx2048m
-XX:MaxPermSize=250m -XX:+UseConcMarkSweepGC
-Dosmoze.path=/tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/Osmoze
-Dspring.profiles.active=webServer -jar
/tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/heavenize-modules.jar
	|- 4093 4073 4069 4069 (sh) 0 0 4550656 164 /bin/sh
/tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/startup.sh
	|- 4073 4069 4069 4069 (java) 249 34 1577267200 24239
/usr/lib/jvm/java-7-openjdk-amd64/bin/java
com.heavenize.yarn.task.ModulesManager -containerId
container_1434471275225_0007_01_000002 -port 5369 -exe
hdfs://hadoop-server/user/hadoop/heavenize/heavenize-modules.jar -conf
hdfs://hadoop-server/user/hadoop/heavenize/config.zip
	|- 4069 1884 4069 4069 (bash) 0 0 12730368 304 /bin/bash -c
/usr/lib/jvm/java-7-openjdk-amd64/bin/java
com.heavenize.yarn.task.ModulesManager -containerId
container_1434471275225_0007_01_000002 -port 5369 -exe
hdfs://hadoop-server/user/hadoop/heavenize/heavenize-modules.jar -conf
hdfs://hadoop-server/user/hadoop/heavenize/config.zip 1>
/usr/local/hadoop/logs/userlogs/application_1434471275225_0007/container_1434471275225_0007_01_000002/stdout
2> /usr/local/hadoop/logs/userlogs/application_1434471275225_0007/container_1434471275225_0007_01_000002/stderr


I don't see any memory excess, any idea where this error comes from?
There is no errors in the container, it just stop logging as a result of
being killed.

--001a11475d4cc26aa70518b54932
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi, I&#39;ve a YARN application that submits containers. I=
n the AplicationMaster logs I see that the container is killed. Here is the=
 logs:<div><br><div><pre style=3D"margin-top:0px;margin-bottom:0px;border:0=
px">Jun 17, 2015 1:31:27 PM com.heavenize.modules.RMCallbackHandler onConta=
inersCompleted
INFO: container &#39;container_1434471275225_0007_01_000002&#39; status is =
ContainerStatus: [ContainerId: container_1434471275225_0007_01_000002, Stat=
e: COMPLETE, Diagnostics: Container [pid=3D4069,containerID=3Dcontainer_143=
4471275225_0007_01_000002] is running beyond virtual memory limits. Current=
 usage: 576.2 MB of 2 GB physical memory used; 4.2 GB of 4.2 GB virtual mem=
ory used. Killing container.
Dump of the process-tree for container_1434471275225_0007_01_000002 :
	|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILL=
IS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
	|- 4094 4093 4069 4069 (java) 2932 94 2916065280 122804 /usr/lib/jvm/java-=
7-openjdk-amd64/bin/java -Xms512m -Xmx2048m -XX:MaxPermSize=3D250m -XX:+Use=
ConcMarkSweepGC -Dosmoze.path=3D/tmp/hadoop-hadoop/nm-local-dir/usercache/h=
adoop/appcache/container_1434471275225_0007_01_000002/Osmoze -Dspring.profi=
les.active=3DwebServer -jar /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoo=
p/appcache/container_1434471275225_0007_01_000002/heavenize-modules.jar=20
	|- 4093 4073 4069 4069 (sh) 0 0 4550656 164 /bin/sh /tmp/hadoop-hadoop/nm-=
local-dir/usercache/hadoop/appcache/container_1434471275225_0007_01_000002/=
startup.sh=20
	|- 4073 4069 4069 4069 (java) 249 34 1577267200 24239 /usr/lib/jvm/java-7-=
openjdk-amd64/bin/java com.heavenize.yarn.task.ModulesManager -containerId =
container_1434471275225_0007_01_000002 -port 5369 -exe hdfs://hadoop-server=
/user/hadoop/heavenize/heavenize-modules.jar -conf hdfs://hadoop-server/use=
r/hadoop/heavenize/config.zip=20
	|- 4069 1884 4069 4069 (bash) 0 0 12730368 304 /bin/bash -c /usr/lib/jvm/j=
ava-7-openjdk-amd64/bin/java com.heavenize.yarn.task.ModulesManager -contai=
nerId container_1434471275225_0007_01_000002 -port 5369 -exe hdfs://hadoop-=
server/user/hadoop/heavenize/heavenize-modules.jar -conf hdfs://hadoop-serv=
er/user/hadoop/heavenize/config.zip 1&gt; /usr/local/hadoop/logs/userlogs/a=
pplication_1434471275225_0007/container_1434471275225_0007_01_000002/stdout=
 2&gt; /usr/local/hadoop/logs/userlogs/application_1434471275225_0007/conta=
iner_1434471275225_0007_01_000002/stderr=20
</pre></div><div><br></div><div>I don&#39;t see any memory excess, any idea=
 where this error comes from?</div></div><div>There is no errors in the con=
tainer, it just stop logging as a result of being killed.</div></div>

--001a11475d4cc26aa70518b54932--