Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B707110484 for ; Mon, 9 Dec 2013 10:04:44 +0000 (UTC) Received: (qmail 99417 invoked by uid 500); 9 Dec 2013 10:04:38 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 98883 invoked by uid 500); 9 Dec 2013 10:04:35 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 98875 invoked by uid 99); 9 Dec 2013 10:04:35 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Dec 2013 10:04:35 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of cnweike@gmail.com designates 74.125.82.52 as permitted sender) Received: from [74.125.82.52] (HELO mail-wg0-f52.google.com) (74.125.82.52) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Dec 2013 10:04:29 +0000 Received: by mail-wg0-f52.google.com with SMTP id x13so3174665wgg.19 for ; Mon, 09 Dec 2013 02:04:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=0oqqQ0cxZ7SP9N/BKmCfnPPIfQ5g8pMnL5RAxcSuAFw=; b=kNQEdgu1c84Khw6ALlZVkHSga3M9YnLTZak6snMm8QYNK8xzTYlYyR3qAurGl3UWGu ZZ2GDs6Lh8Z+HcyA68QtrwiXKDBkg0Q1752QKjPDH7iAnQycMsQ/KhgXqzaJNGnmr2tO AeAyjJ8mFIah8IuDcTvvf5o+vY7ILsYoBmq1HrXZsWCQ4fftrzuMoNt/tKk/yWdXGWbv qdeEYd4VZ3d0eVONMuD0UWyttydNl3A/LQJVpxDo4OnXUnMGHk70MaDX7iFT9QTn7tEN r/SIGdSyDTYASKvrdokD7TZLeDMgVv83TEbsQyt0jzXJeS+U/OH+gQe6p2/UiLPg8HA+ 9SPA== MIME-Version: 1.0 X-Received: by 10.194.243.170 with SMTP id wz10mr2792572wjc.74.1386583449135; Mon, 09 Dec 2013 02:04:09 -0800 (PST) Received: by 10.194.41.74 with HTTP; Mon, 9 Dec 2013 02:04:09 -0800 (PST) In-Reply-To: References: Date: Mon, 9 Dec 2013 18:04:09 +0800 Message-ID: Subject: Re: Container [pid=22885,containerID=container_1386156666044_0001_01_000013] is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical memory used; 332.5 GB of 8 GB virtual memory used. Killing container. From: panfei To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=089e0149411aeff3d604ed171dc1 X-Virus-Checked: Checked by ClamAV on apache.org --089e0149411aeff3d604ed171dc1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi All, thanks for your replies, I have found the point: there is a Memory Leak(open a file for each record) in the Hive UDF method, after fixing it, everything goes well now. 2013/12/6 Vinod Kumar Vavilapalli > Something looks really bad on your cluster. The JVM's heap size is 200MB > but its virtual memory has ballooned to a monstrous 332GB. Does that ring > any bell? Can you run regular java applications on this node? This doesn'= t > seem related to YARN per-se. > > +Vinod > Hortonworks Inc. > http://hortonworks.com/ > > > On Wed, Dec 4, 2013 at 5:16 AM, panfei wrote: > >> >> >> ---------- Forwarded message ---------- >> From: panfei >> Date: 2013/12/4 >> Subject: Container >> [pid=3D22885,containerID=3Dcontainer_1386156666044_0001_01_000013] is ru= nning >> beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical >> memory used; 332.5 GB of 8 GB virtual memory used. Killing container. >> To: CDH Users >> >> >> Hi All: >> >> We are using CDH4.5 Hadoop for our production, when submit some (not all= ) >> jobs from hive, we get the following exception info , seems the physical >> memory and virtual memory both not enough for the job to run: >> >> >> Task with the most failures(4): >> ----- >> Task ID: >> task_1386156666044_0001_m_000000 >> >> URL: >> >> http://namenode-1:8088/taskdetails.jsp?jobid=3Djob_1386156666044_0001&ti= pid=3Dtask_1386156666044_0001_m_000000 >> ----- >> Diagnostic Messages for this Task: >> Container [pid=3D22885,containerID=3Dcontainer_1386156666044_0001_01_000= 013] >> is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB >> physical memory used; 332.5 GB of 8 GB virtual memory used. Killing >> container. >> Dump of the process-tree for container_1386156666044_0001_01_000013 : >> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) >> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE >> |- 22885 22036 22885 22885 (java) 5414 108 356993519616 271953 >> /usr/java/default/bin/java -Djava.net.preferIPv4Stack=3Dtrue >> -Dhadoop.metrics.log.level=3DWARN -Xmx200m >> -Djava.io.tmpdir=3D/data/yarn/local/usercache/hive/appcache/application_= 1386156666044_0001/container_1386156666044_0001_01_000013/tmp >> -Dlog4j.configuration=3Dcontainer-log4j.properties >> -Dyarn.app.mapreduce.container.log.dir=3D/var/log/hadoop-yarn/containers= /application_1386156666044_0001/container_1386156666044_0001_01_000013 >> -Dyarn.app.mapreduce.container.log.filesize=3D0 -Dhadoop.root.logger=3DI= NFO,CLA >> org.apache.hadoop.mapred.YarnChild 192.168.101.55 60841 >> attempt_1386156666044_0001_m_000000_3 13 >> >> following is some of our configuration: >> >> >> yarn.nodemanager.resource.memory-mb >> 12288 >> >> >> >> yarn.nodemanager.vmem-pmem-ratio >> 8 >> >> >> >> yarn.nodemanager.vmem-check-enabled >> false >> >> >> >> yarn.nodemanager.resource.cpu-vcores >> 6 >> >> >> can you give me some advice? thanks a lot. >> -- >> =E4=B8=8D=E5=AD=A6=E4=B9=A0=EF=BC=8C=E4=B8=8D=E7=9F=A5=E9=81=93 >> >> >> >> -- >> =E4=B8=8D=E5=AD=A6=E4=B9=A0=EF=BC=8C=E4=B8=8D=E7=9F=A5=E9=81=93 >> > > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity > to which it is addressed and may contain information that is confidential= , > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified th= at > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediate= ly > and delete it from your system. Thank You. --=20 =E4=B8=8D=E5=AD=A6=E4=B9=A0=EF=BC=8C=E4=B8=8D=E7=9F=A5=E9=81=93 --089e0149411aeff3d604ed171dc1 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi All, thanks for your replies, I have found the point: t= here is a Memory Leak(open a file for each record) in the Hive UDF method, = after fixing it, everything goes well now.


2013/12/6 Vinod Kumar Vavilapalli <vinodkv@hortonworks.com>
Something looks really bad on your cluster. The JVM's = heap size is 200MB but its virtual memory has ballooned to a monstrous 332G= B. Does that ring any bell? Can you run regular java applications on this n= ode? This doesn't seem related to YARN per-se.

+Vinod
Hortonworks Inc= .
http://hortonwor= ks.com/


On Wed, Dec 4, 2013 at 5:16 AM, panfei <= span dir=3D"ltr"><cnweike@gmail.com> wrote:


---------- Forwarded me= ssage ----------
From: panfei <cnweik= e@gmail.com>
Date: 2013/12/4
Subject: Container [pid=3D22885,containerID=3Dcontainer_= 1386156666044_0001_01_000013] is running beyond physical memory limits. Cur= rent usage: 1.0 GB of 1 GB physical memory used; 332.5 GB of 8 GB virtual m= emory used. Killing container.
To: CDH Users <cdh-user@cloudera.org>


Hi All:
We are using CDH4.5 Hadoop for our production, when submit some = (not all) jobs from hive, we get the following exception info , seems the p= hysical memory and virtual memory both not enough for the job to run:


Task with the most failures(4):
-----
Task ID:
=C2=A0 tas= k_1386156666044_0001_m_000000

URL:
=C2=A0 http://namenode-1:8088/taskd= etails.jsp?jobid=3Djob_1386156666044_0001&tipid=3Dtask_1386156666044_00= 01_m_000000
-----
Diagnostic Messages for this Task:
Container [pid=3D22885,conta= inerID=3Dcontainer_1386156666044_0001_01_000013] is running beyond physical= memory limits. Current usage: 1.0 GB of 1 GB physical memory used; 332.5 G= B of 8 GB virtual memory used. Killing container.
Dump of the process-tree for container_1386156666044_0001_01_000013 :
= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |- PID PPID PGRPID SESSID CMD_NA= ME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAG= E(PAGES) FULL_CMD_LINE
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |- 228= 85 22036 22885 22885 (java) 5414 108 356993519616 271953 /usr/java/default/= bin/java -Djava.net.preferIPv4Stack=3Dtrue -Dhadoop.metrics.log.level=3DWAR= N -Xmx200m -Djava.io.tmpdir=3D/data/yarn/local/usercache/hive/appcache/appl= ication_1386156666044_0001/container_1386156666044_0001_01_000013/tmp -Dlog= 4j.configuration=3Dcontainer-log4j.properties -Dyarn.app.mapreduce.containe= r.log.dir=3D/var/log/hadoop-yarn/containers/application_1386156666044_0001/= container_1386156666044_0001_01_000013 -Dyarn.app.mapreduce.container.log.f= ilesize=3D0 -Dhadoop.root.logger=3DINFO,CLA org.apache.hadoop.mapred.YarnCh= ild 192.168.101.55 60841 attempt_1386156666044_0001_m_000000_3 13

following is some of our configuration:

=C2= =A0 <property>
=C2=A0=C2=A0=C2=A0 <name>yarn.nodemanager.res= ource.memory-mb</name>
=C2=A0=C2=A0=C2=A0 <value>12288</v= alue>
=C2=A0 </property>

=C2=A0 <property>
=C2=A0=C2=A0=C2=A0 <name>yarn.nodemana= ger.vmem-pmem-ratio</name>
=C2=A0=C2=A0=C2=A0 <value>8</v= alue>
=C2=A0 </property>

=C2=A0 <property>
=C2= =A0=C2=A0=C2=A0 <name>yarn.nodemanager.vmem-check-enabled</name>= ;
=C2=A0=C2=A0=C2=A0 <value>false</value>
=C2=A0 </property= >

=C2=A0 <property>
=C2=A0=C2=A0=C2=A0 <name>yarn.= nodemanager.resource.cpu-vcores</name>
=C2=A0=C2=A0=C2=A0 <valu= e>6</value>
=C2=A0 </property>

can you give me some advice? thanks a lot.
--
=E4=B8=8D=E5=AD=A6=E4=B9=A0=EF=BC=8C=E4=B8=8D=E7=9F=A5=E9=81=93


--
=E4=B8=8D=E5=AD=A6=E4=B9=A0=EF=BC=8C=E4=B8=8D=E7=9F=A5=E9=81=93


CONFIDENTIALITY NOTICE
NOTICE: This message is = intended for the use of the individual or entity to which it is addressed a= nd may contain information that is confidential, privileged and exempt from= disclosure under applicable law. If the reader of this message is not the = intended recipient, you are hereby notified that any printing, copying, dis= semination, distribution, disclosure or forwarding of this communication is= strictly prohibited. If you have received this communication in error, ple= ase contact the sender immediately and delete it from your system. Thank Yo= u.



--
=E4=B8=8D=E5=AD=A6=E4=B9=A0=EF=BC=8C= =E4=B8=8D=E7=9F=A5=E9=81=93
--089e0149411aeff3d604ed171dc1--