Return-Path: X-Original-To: apmail-apex-users-archive@minotaur.apache.org Delivered-To: apmail-apex-users-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 63B7918F86 for ; Fri, 11 Mar 2016 15:04:05 +0000 (UTC) Received: (qmail 9611 invoked by uid 500); 11 Mar 2016 15:04:05 -0000 Delivered-To: apmail-apex-users-archive@apex.apache.org Received: (qmail 9494 invoked by uid 500); 11 Mar 2016 15:04:04 -0000 Mailing-List: contact users-help@apex.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@apex.incubator.apache.org Delivered-To: mailing list users@apex.incubator.apache.org Received: (qmail 9485 invoked by uid 99); 11 Mar 2016 15:04:04 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2016 15:04:04 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 818B7180522 for ; Fri, 11 Mar 2016 15:04:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -2.349 X-Spam-Level: X-Spam-Status: No, score=-2.349 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=2, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-0.329] autolearn=disabled Received: from mx2-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id gtJIhFikeyLT for ; Fri, 11 Mar 2016 15:04:03 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx2-lw-us.apache.org (ASF Mail Server at mx2-lw-us.apache.org) with SMTP id 6D5F95F1BE for ; Fri, 11 Mar 2016 15:04:02 +0000 (UTC) Received: (qmail 9479 invoked by uid 99); 11 Mar 2016 15:04:01 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2016 15:04:01 +0000 Received: from mail-vk0-f53.google.com (mail-vk0-f53.google.com [209.85.213.53]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 81D5C1A031C for ; Fri, 11 Mar 2016 15:04:01 +0000 (UTC) Received: by mail-vk0-f53.google.com with SMTP id e6so137016311vkh.2 for ; Fri, 11 Mar 2016 07:04:01 -0800 (PST) X-Gm-Message-State: AD7BkJIcNBOUAcZZpJTn86NBi46REltIY7/oiXTbgII7BoaZ5LmGPFnDKXF4I+uzxi1kRuRgQicFi0fosCSoR84o MIME-Version: 1.0 X-Received: by 10.31.182.143 with SMTP id g137mr10187626vkf.45.1457708640409; Fri, 11 Mar 2016 07:04:00 -0800 (PST) Received: by 10.159.41.3 with HTTP; Fri, 11 Mar 2016 07:04:00 -0800 (PST) Date: Fri, 11 Mar 2016 20:34:00 +0530 X-Gmail-Original-Message-ID: Message-ID: Subject: AWS EMR: Container is running beyond virtual memory limits From: "Pradeep A. Dalvi" To: users@apex.incubator.apache.org Content-Type: multipart/alternative; boundary=001a114386acb2c712052dc73d98 --001a114386acb2c712052dc73d98 Content-Type: text/plain; charset=UTF-8 We are facing following error message while starting any containers on AWS EMR. Container [pid=8107,containerID=container_1457702160744_0001_01_000007] is running beyond virtual memory limits. Current usage: 186.1 MB of 256 MB physical memory used; 2.0 GB of 1.3 GB virtual memory used. Killing container. Dump of the process-tree for container_1457702160744_0001_01_000007 : |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE |- 8222 8107 8107 8107 (java) 589 62 2041503744 46944 /usr/lib/jvm/java-openjdk/bin/java -Xmx234881024 -Ddt.attr.APPLICATION_PATH=hdfs://ip-172-31-9-174.ec2.internal:8020/user/hadoop/datatorrent/apps/application_1457702160744_0001 -Djava.io.tmpdir=/mnt1/yarn/usercache/hadoop/appcache/application_1457702160744_0001/container_1457702160744_0001_01_000007/tmp -Ddt.cid=container_1457702160744_0001_01_000007 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=/var/log/hadoop-yarn/containers/application_1457702160744_0001/container_1457702160744_0001_01_000007 com.datatorrent.stram.engine.StreamingContainer |- 8107 8105 8107 8107 (bash) 1 5 115806208 705 /bin/bash -c /usr/lib/jvm/java-openjdk/bin/java -Xmx234881024 -Ddt.attr.APPLICATION_PATH=hdfs://ip-172-31-9-174.ec2.internal:8020/user/hadoop/datatorrent/apps/application_1457702160744_0001 -Djava.io.tmpdir=/mnt1/yarn/usercache/hadoop/appcache/application_1457702160744_0001/container_1457702160744_0001_01_000007/tmp -Ddt.cid=container_1457702160744_0001_01_000007 -Dhadoop.root.logger=INFO,RFA -Dhadoop.log.dir=/var/log/hadoop-yarn/containers/application_1457702160744_0001/container_1457702160744_0001_01_000007 com.datatorrent.stram.engine.StreamingContainer 1>/var/log/hadoop-yarn/containers/application_1457702160744_0001/container_1457702160744_0001_01_000007/stdout 2>/var/log/hadoop-yarn/containers/application_1457702160744_0001/container_1457702160744_0001_01_000007/stderr Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 We had 1 m3.xlarge MASTER & 2 m3.xlarge CORE instances provisioned. We also have tried m4.4xlarge instances. EMR Task configurations can be found at http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/TaskConfiguration_H2.html We tried changing following yarn configurations, however they did not seem to help much. *yarn.nodemanager.resource.memory-mb**12288* *yarn.scheduler.maximum-allocation-mb**4096* *yarn.nodemanager.vmem-check-enabled* *false* *yarn.nodemanager.vmem-pmem-ratio* *50* Thanks, -- Pradeep A. Dalvi --001a114386acb2c712052dc73d98 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
We are facing following error message while starting = any containers on AWS EMR.

Containe=
r [pid=3D8107,containerID=3Dcontainer_1457702160744_0001_01_000007] is runn=
ing beyond virtual memory limits. Current usage: 186.1 MB of 256 MB physica=
l memory used; 2.0 GB of 1.3 GB virtual memory used. Killing container.
Dump of the process-tree for container_1457702160744_0001_01_000007 :
	|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILL=
IS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
	|- 8222 8107 8107 8107 (java) 589 62 2041503744 46944 /usr/lib/jvm/java-op=
enjdk/bin/java -Xmx234881024 -Ddt.attr.APPLICATION_PATH=3Dhdfs://ip-172-31-=
9-174.ec2.internal:8020/user/hadoop/datatorrent/apps/application_1457702160=
744_0001 -Djava.io.tmpdir=3D/mnt1/yarn/usercache/hadoop/appcache/applicatio=
n_1457702160744_0001/container_1457702160744_0001_01_000007/tmp -Ddt.cid=3D=
container_1457702160744_0001_01_000007 -Dhadoop.root.logger=3DINFO,RFA -Dha=
doop.log.dir=3D/var/log/hadoop-yarn/containers/application_1457702160744_00=
01/container_1457702160744_0001_01_000007 com.datatorrent.stram.engine.Stre=
amingContainer=20
	|- 8107 8105 8107 8107 (bash) 1 5 115806208 705 /bin/bash -c /usr/lib/jvm/=
java-openjdk/bin/java  -Xmx234881024  -Ddt.attr.APPLICATION_PATH=3Dhdfs://i=
p-172-31-9-174.ec2.internal:8020/user/hadoop/datatorrent/apps/application_1=
457702160744_0001 -Djava.io.tmpdir=3D/mnt1/yarn/usercache/hadoop/appcache/a=
pplication_1457702160744_0001/container_1457702160744_0001_01_000007/tmp -D=
dt.cid=3Dcontainer_1457702160744_0001_01_000007 -Dhadoop.root.logger=3DINFO=
,RFA -Dhadoop.log.dir=3D/var/log/hadoop-yarn/containers/application_1457702=
160744_0001/container_1457702160744_0001_01_000007 com.datatorrent.stram.en=
gine.StreamingContainer 1>/var/log/hadoop-yarn/containers/application_14=
57702160744_0001/container_1457702160744_0001_01_000007/stdout 2>/var/lo=
g/hadoop-yarn/containers/application_1457702160744_0001/container_145770216=
0744_0001_01_000007/stderr  =20

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

We had 1 m3.xlarge MASTER & 2 m3.xlarge CORE instances pro= visioned. We also have tried m4.4xlarge instances. EMR Task configurations = can be found at http://docs.aws.amazon.com/Ela= sticMapReduce/latest/DeveloperGuide/TaskConfiguration_H2.html
=

We tried changing following yarn configurations, howeve= r they did not seem to help much.

=C2=A0 <prope= rty><name>yarn.nodemanager.resource.memory-mb</name>&= lt;value>12288</value></property>
=C2=A0 &l= t;property><name>yarn.scheduler.maximum-allocation-mb</n= ame><value>4096</value></property>
<= div>=C2=A0 <property> <name>yarn.nodemanager.vmem-check-enab= led</name> <value>false</value> </property&= gt;
=C2=A0 <property> <name>yarn.nodemanager.vmem-= pmem-ratio</name> <value>50</value></proper= ty>


Thanks,
--
Pradeep A. Dalvi
--001a114386acb2c712052dc73d98--