Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id EA26010933 for ; Thu, 5 Dec 2013 08:38:56 +0000 (UTC) Received: (qmail 18292 invoked by uid 500); 5 Dec 2013 08:38:42 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 17638 invoked by uid 500); 5 Dec 2013 08:38:38 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 17623 invoked by uid 99); 5 Dec 2013 08:38:36 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Dec 2013 08:38:36 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of cnweike@gmail.com designates 74.125.82.172 as permitted sender) Received: from [74.125.82.172] (HELO mail-we0-f172.google.com) (74.125.82.172) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Dec 2013 08:38:32 +0000 Received: by mail-we0-f172.google.com with SMTP id w62so10640285wes.31 for ; Thu, 05 Dec 2013 00:38:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=UId/qdfNreiEeVNyZDD7OfZpQUHw0xrMwAN4HV8nNjY=; b=jfIVwvGn/O1feOCw43jhM66gcBerCsZaiR1D8isbx8vMm/cM577ObhPt1IXk1C3a0b YXZkhu2R5ZcsNB/pvjmnhgI0iq4hCwcP0uoPtXYpKkB61GYm5VyMYvghW3Qd+HrFPO0z jCT1fnr08OCALIyZz6LoRLf2miim4qH2Pg4EksviDmcIpp0aW85EFacZsrE6VCH88pMO 4H/7b2yY1nHZOYSt1Hg4XxisgrHthpmn0FJ8UWrIZsFShe1uTkSsMH5KfDxhGdfwEy7/ CKHoT0uSsI8Ay1wEie852FNjSk6HSzeAqK1BYg3jnPs+WyT6LivjXJ2J0uWnkv3+GuDa 9Agg== MIME-Version: 1.0 X-Received: by 10.194.60.73 with SMTP id f9mr16182353wjr.65.1386232691179; Thu, 05 Dec 2013 00:38:11 -0800 (PST) Received: by 10.194.41.74 with HTTP; Thu, 5 Dec 2013 00:38:11 -0800 (PST) In-Reply-To: References: Date: Thu, 5 Dec 2013 16:38:11 +0800 Message-ID: Subject: Re: Container [pid=22885,containerID=container_1386156666044_0001_01_000013] is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical memory used; 332.5 GB of 8 GB virtual memory used. Killing container. From: panfei To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=047d7b86de822247a104ecc57345 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b86de822247a104ecc57345 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi YouPeng, thanks for your advice. I have read the docs and configure the parameters as follows: Physical Server: 8 cores CPU, 16GB memory. For YARN: yarn.nodemanager.resource.memory-mb set to 12GB and keep 4GB for the OS. yarn.scheduler.minimum-allocation-mb set to 2048M as the minimum allocation unit for the container. yarn.nodemanager.vmem-pmem-ratio is the default value 2.1 FOR MAPREDUCE: mapreduce.map.memory.mb set to 2048 for map task containers. mapreduce.reduce.memory.mb set to 4096 for reduce task containers. mapreduce.map.java.opts set to -Xmx1536m mapreduce.reduce.java.opts set to -Xmx3072m after setting theses parameters, the problem still there, I think it's time to get back to HADOOP 1.0 infrastructure. thanks for your advice again. 2013/12/5 YouPeng Yang > Hi > > please reference to > http://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/ > > > > 2013/12/5 panfei > >> we have already tried several values of these two parameters, but it >> seems no use. >> >> >> 2013/12/5 Tsuyoshi OZAWA >> >>> Hi, >>> >>> Please check the properties like mapreduce.reduce.memory.mb and >>> mapredce.map.memory.mb in mapred-site.xml. These properties decide >>> resource limits for mappers/reducers. >>> >>> On Wed, Dec 4, 2013 at 10:16 PM, panfei wrote: >>> > >>> > >>> > ---------- Forwarded message ---------- >>> > From: panfei >>> > Date: 2013/12/4 >>> > Subject: Container >>> > [pid=3D22885,containerID=3Dcontainer_1386156666044_0001_01_000013] is >>> running >>> > beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical >>> memory >>> > used; 332.5 GB of 8 GB virtual memory used. Killing container. >>> > To: CDH Users >>> > >>> > >>> > Hi All: >>> > >>> > We are using CDH4.5 Hadoop for our production, when submit some (not >>> all) >>> > jobs from hive, we get the following exception info , seems the >>> physical >>> > memory and virtual memory both not enough for the job to run: >>> > >>> > >>> > Task with the most failures(4): >>> > ----- >>> > Task ID: >>> > task_1386156666044_0001_m_000000 >>> > >>> > URL: >>> > >>> > >>> http://namenode-1:8088/taskdetails.jsp?jobid=3Djob_1386156666044_0001&t= ipid=3Dtask_1386156666044_0001_m_000000 >>> > ----- >>> > Diagnostic Messages for this Task: >>> > Container >>> [pid=3D22885,containerID=3Dcontainer_1386156666044_0001_01_000013] is >>> > running beyond physical memory limits. Current usage: 1.0 GB of 1 GB >>> > physical memory used; 332.5 GB of 8 GB virtual memory used. Killing >>> > container. >>> > Dump of the process-tree for container_1386156666044_0001_01_000013 : >>> > |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) >>> > SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LI= NE >>> > |- 22885 22036 22885 22885 (java) 5414 108 356993519616 27195= 3 >>> > /usr/java/default/bin/java -Djava.net.preferIPv4Stack=3Dtrue >>> > -Dhadoop.metrics.log.level=3DWARN -Xmx200m >>> > >>> -Djava.io.tmpdir=3D/data/yarn/local/usercache/hive/appcache/application= _1386156666044_0001/container_1386156666044_0001_01_000013/tmp >>> > -Dlog4j.configuration=3Dcontainer-log4j.properties >>> > >>> -Dyarn.app.mapreduce.container.log.dir=3D/var/log/hadoop-yarn/container= s/application_1386156666044_0001/container_1386156666044_0001_01_000013 >>> > -Dyarn.app.mapreduce.container.log.filesize=3D0 >>> -Dhadoop.root.logger=3DINFO,CLA >>> > org.apache.hadoop.mapred.YarnChild 192.168.101.55 60841 >>> > attempt_1386156666044_0001_m_000000_3 13 >>> > >>> > following is some of our configuration: >>> > >>> > >>> > yarn.nodemanager.resource.memory-mb >>> > 12288 >>> > >>> > >>> > >>> > yarn.nodemanager.vmem-pmem-ratio >>> > 8 >>> > >>> > >>> > >>> > yarn.nodemanager.vmem-check-enabled >>> > false >>> > >>> > >>> > >>> > yarn.nodemanager.resource.cpu-vcores >>> > 6 >>> > >>> > >>> > can you give me some advice? thanks a lot. >>> > -- >>> > =E4=B8=8D=E5=AD=A6=E4=B9=A0=EF=BC=8C=E4=B8=8D=E7=9F=A5=E9=81=93 >>> > >>> > >>> > >>> > -- >>> > =E4=B8=8D=E5=AD=A6=E4=B9=A0=EF=BC=8C=E4=B8=8D=E7=9F=A5=E9=81=93 >>> >>> >>> >>> -- >>> - Tsuyoshi >>> >> >> >> >> -- >> =E4=B8=8D=E5=AD=A6=E4=B9=A0=EF=BC=8C=E4=B8=8D=E7=9F=A5=E9=81=93 >> > > --=20 =E4=B8=8D=E5=AD=A6=E4=B9=A0=EF=BC=8C=E4=B8=8D=E7=9F=A5=E9=81=93 --047d7b86de822247a104ecc57345 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi YouPeng, thanks for your advice. I = have read the docs and configure the parameters as follows:

Ph= ysical Server: 8 cores CPU, 16GB memory.

For YARN:

yarn.nodemanager.resource.memory-mb set to 12GB and keep 4GB for the= OS.

yarn.scheduler.minimum-allocation-mb set to 204= 8M=C2=A0 as the minimum allocation unit for the container.

yarn.nodemanager.vmem-pmem-ratio is the def= ault value 2.1


FOR MAPREDUCE:

<= code class=3D"">mapreduce.map.memory.mb set to 2048 for map task containers= .

mapreduce.reduce.memory.mb set to 4096 for= reduce task containers.

mapreduce.map.java.opts set to -Xmx1536m

mapreduce.reduce.java.o= pts set to -Xmx3072m



after set= ting theses parameters, the problem still there, I think it's time to g= et back to HADOOP 1.0 infrastructure.

thanks for your advice again.
<= div>



2013/12/5 YouPeng Yang <yypvsxf19870706@gmail.com>


2013/12/5 pan= fei <cnweike@gmail.com>
we have already tried several values of these two paramete= rs, but it seems no use.

=
2013/12/5 Tsuyoshi OZAWA &= lt;ozawa.tsuy= oshi@gmail.com>
Hi,

Please check the properties like mapreduce.reduce.memory.mb and
mapredce.map.memory.mb in mapred-site.xml. These properties decide
resource limits for mappers/reducers.

On Wed, Dec 4, 2013 at 10:16 PM, panfei <cnweike@gmail.com> wrote:
>
>
> ---------- Forwarded message ----------
> From: panfei <cnweike@gmail.com>
> Date: 2013/12/4
> Subject: Container
> [pid=3D22885,containerID=3Dcontainer_1386156666044_0001_01_000013] is = running
> beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical = memory
> used; 332.5 GB of 8 GB virtual memory used. Killing container.
> To: CDH Users <cdh-user@cloudera.org>
>
>
> Hi All:
>
> We are using CDH4.5 Hadoop for our production, when submit some (not a= ll)
> jobs from hive, we get the following exception info , seems the physic= al
> memory and virtual memory both not enough for the job to run:
>
>
> Task with the most failures(4):
> -----
> Task ID:
> =C2=A0 task_1386156666044_0001_m_000000
>
> URL:
>
> = http://namenode-1:8088/taskdetails.jsp?jobid=3Djob_1386156666044_0001&t= ipid=3Dtask_1386156666044_0001_m_000000
> -----
> Diagnostic Messages for this Task:
> Container [pid=3D22885,containerID=3Dcontainer_1386156666044_0001_01_0= 00013] is
> running beyond physical memory limits. Current usage: 1.0 GB of 1 GB > physical memory used; 332.5 GB of 8 GB virtual memory used. Killing > container.
> Dump of the process-tree for container_1386156666044_0001_01_000013 :<= br> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 |- PID PPID PGRPID SESSID CMD_NAME USER_MO= DE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LIN= E
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 |- 22885 22036 22885 22885 (java) 5414 108= 356993519616 271953
> /usr/java/default/bin/java -Djava.net.preferIPv4Stack=3Dtrue
> -Dhadoop.metrics.log.level=3DWARN -Xmx200m
> -Djava.io.tmpdir=3D/data/yarn/local/usercache/hive/appcache/applicatio= n_1386156666044_0001/container_1386156666044_0001_01_000013/tmp
> -Dlog4j.configuration=3Dcontainer-log4j.properties
> -Dyarn.app.mapreduce.container.log.dir=3D/var/log/hadoop-yarn/containe= rs/application_1386156666044_0001/container_1386156666044_0001_01_000013 > -Dyarn.app.mapreduce.container.log.filesize=3D0 -Dhadoop.root.logger= =3DINFO,CLA
> org.apache.hadoop.mapred.YarnChild 192.168.101.55 60841
> attempt_1386156666044_0001_m_000000_3 13
>
> following is some of our configuration:
>
> =C2=A0 <property>
> =C2=A0 =C2=A0 <name>yarn.nodemanager.resource.memory-mb</name= >
> =C2=A0 =C2=A0 <value>12288</value>
> =C2=A0 </property>
>
> =C2=A0 <property>
> =C2=A0 =C2=A0 <name>yarn.nodemanager.vmem-pmem-ratio</name>= ;
> =C2=A0 =C2=A0 <value>8</value>
> =C2=A0 </property>
>
> =C2=A0 <property>
> =C2=A0 =C2=A0 <name>yarn.nodemanager.vmem-check-enabled</name= >
> =C2=A0 =C2=A0 <value>false</value>
> =C2=A0 </property>
>
> =C2=A0 <property>
> =C2=A0 =C2=A0 <name>yarn.nodemanager.resource.cpu-vcores</nam= e>
> =C2=A0 =C2=A0 <value>6</value>
> =C2=A0 </property>
>
> can you give me some advice? thanks a lot.
> --
> =E4=B8=8D=E5=AD=A6=E4=B9=A0=EF=BC=8C=E4=B8=8D=E7=9F=A5=E9=81=93
>
>
>
> --
> =E4=B8=8D=E5=AD=A6=E4=B9=A0=EF=BC=8C=E4=B8=8D=E7=9F=A5=E9=81=93



--
- Tsuyoshi



--
=E4=B8=8D=E5=AD=A6=E4=B9=A0=EF=BC=8C=E4=B8= =8D=E7=9F=A5=E9=81=93




--
=E4=B8=8D= =E5=AD=A6=E4=B9=A0=EF=BC=8C=E4=B8=8D=E7=9F=A5=E9=81=93
--047d7b86de822247a104ecc57345--