Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CF128177EE for ; Mon, 23 Mar 2015 05:10:48 +0000 (UTC) Received: (qmail 5991 invoked by uid 500); 23 Mar 2015 05:10:43 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 5888 invoked by uid 500); 23 Mar 2015 05:10:43 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 5877 invoked by uid 99); 23 Mar 2015 05:10:43 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Mar 2015 05:10:43 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW X-Spam-Check-By: apache.org Received-SPF: error (athena.apache.org: local policy) Received: from [209.85.218.46] (HELO mail-oi0-f46.google.com) (209.85.218.46) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Mar 2015 05:10:38 +0000 Received: by oiag65 with SMTP id g65so132015684oia.2 for ; Sun, 22 Mar 2015 22:09:57 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=OxBgHM1hd2VSPyh9GvuS4zeLNkQseyDAWSMn7HkEE1c=; b=Yz+3Cm95e3HvSd38kmCLDSy8dkHd8c9uE9uju6gFNzBn8L54y20KinqSNpGig/6DIt OciT+kDUfp7Gmk8HRdVaC9bps5FaWRdxCT2cQyA0z4kx2BykMAcLK5bOwVNxe3/mYYTD tyLHMV32e6IdmsE++nqmuG252vmW6bRK6U+7/gQevtircS8w6RJPRtEAcVlT/T+L8XrQ tZs/Q3FG/QoxBmKoYGjstgvHZKMO+GgIzfEY2svm06YxUWvTYgEdQMV3n4XCIPG/q+uM BBosmewxGtYlre7n2wrhxt22uk+RAi3a10NjGcy/ruOQk4jluVkfjZt7Tuph7lZ6dLB8 bsIQ== X-Gm-Message-State: ALoCoQmH1H8x9AE1bD2TjNvHxjiCA97C9v2SgynuROXUyeAyMV29jHx+WrnMqpVlgGkPEJybUjgh MIME-Version: 1.0 X-Received: by 10.60.150.129 with SMTP id ui1mr74295731oeb.10.1427087397120; Sun, 22 Mar 2015 22:09:57 -0700 (PDT) Received: by 10.182.38.132 with HTTP; Sun, 22 Mar 2015 22:09:57 -0700 (PDT) In-Reply-To: <8CD6759A-D28A-49BD-B7D6-66F1D1589182@gmail.com> References: <8CD6759A-D28A-49BD-B7D6-66F1D1589182@gmail.com> Date: Mon, 23 Mar 2015 14:09:57 +0900 Message-ID: Subject: Re: Container beyond virtual memory limits From: =?UTF-8?B?RHJha2Xrr7zsmIHqt7w=?= To: user Content-Type: multipart/alternative; boundary=047d7b4513d85e80b70511edadcf X-Virus-Checked: Checked by ClamAV on apache.org --047d7b4513d85e80b70511edadcf Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi, See "6. Killing of Tasks Due to Virtual Memory Usage" in http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-co= nsuming-gotchas/ Drake =EB=AF=BC=EC=98=81=EA=B7=BC Ph.D kt NexR On Sun, Mar 22, 2015 at 12:43 PM, Fei Hu wrote: > Hi, > > I just test my yarn installation, and run a Wordcount program. But it > always report the following error, who knows how to solve it? Thank you i= n > advance. > > Container [pid=3D7954,containerID=3Dcontainer_1426992254950_0002_01_00000= 5] is > running beyond virtual memory limits. Current usage: 13.6 MB of 1 GB > physical memory used; 4.3 GB of 2.1 GB virtual memory used. Killing > container. > Dump of the process-tree for container_1426992254950_0002_01_000005 : > |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) > SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE > |- 7960 7954 7954 7954 (java) 5 0 4576591872 3199 > /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java > -Djava.net.preferIPv4Stack=3Dtrue -Dhadoop.metrics.log.level=3DWARN 1638 > -Djava.io.tmpdir=3D/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/= application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp > -Dlog4j.configuration=3Dcontainer-log4j.properties > -Dyarn.app.container.log.dir=3D/home/hadoop-lzl/hadoop-2.6.0/logs/userlog= s/application_1426992254950_0002/container_1426992254950_0002_01_000005 > -Dyarn.app.container.log.filesize=3D0 -Dhadoop.root.logger=3DINFO,CLA > org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542 > attempt_1426992254950_0002_m_000003_0 5 > |- 7954 7949 7954 7954 (bash) 0 0 65421312 275 /bin/bash -c > /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java > -Djava.net.preferIPv4Stack=3Dtrue -Dhadoop.metrics.log.level=3DWARN 1638 > -Djava.io.tmpdir=3D/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/= application_1426992254950_0002/container_1426992254950_0002_01_000005/tmp > -Dlog4j.configuration=3Dcontainer-log4j.properties > -Dyarn.app.container.log.dir=3D/home/hadoop-lzl/hadoop-2.6.0/logs/userlog= s/application_1426992254950_0002/container_1426992254950_0002_01_000005 > -Dyarn.app.container.log.filesize=3D0 -Dhadoop.root.logger=3DINFO,CLA > org.apache.hadoop.mapred.YarnChild 199.26.254.140 36542 > attempt_1426992254950_0002_m_000003_0 5 > 1>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0= 002/container_1426992254950_0002_01_000005/stdout > 2>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_0= 002/container_1426992254950_0002_01_000005/stderr > > > Exception from container-launch. > Container id: container_1426992254950_0002_01_000005 > Exit code: 1 > Stack trace: ExitCodeException exitCode=3D1: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) > at org.apache.hadoop.util.Shell.run(Shell.java:455) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launch= Container(DefaultContainerExecutor.java:211) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Conta= inerLaunch.call(ContainerLaunch.java:302) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Conta= inerLaunch.call(ContainerLaunch.java:82) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java= :1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav= a:603) > at java.lang.Thread.run(Thread.java:679) > > Thanks, > Fei > --047d7b4513d85e80b70511edadcf Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi,

See "6. Killing of Tasks Due t= o Virtual Memory Usage" in =C2=A0http://b= log.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-= gotchas/=C2=A0

=
= Drake =EB=AF=BC=EC=98=81=EA=B7=BC Ph.D
kt NexR
<= /div>

On Sun, Mar 22, 2015 at 12:43 PM, Fei Hu <h= ufei68@gmail.com> wrote:
Hi,

I just test my yar= n installation, and run a Wordcount program. But it always report the follo= wing error, who knows how to solve it? Thank you in advance.

=
Conta= iner [pid=3D7954,containerID=3Dcontainer_1426992254950_0002_01_000005] is r= unning beyond virtual memory limits. Current usage: 13.6 MB of 1 GB physica= l memory used; 4.3 GB of 2.1 GB virtual memory used= . Killing container.
Dump of the process-tree for container_1426992254950_0002_= 01_000005 :
|- PID PPID PGRPID SESSID CMD= _NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_U= SAGE(PAGES) FULL_CMD_LINE
|- 7960 7954 79= 54 7954 (java) 5 0 4576591872 3199 /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.= x86_64/jre/bin/java -Djava.net.preferIPv4Stack=3Dtrue -Dhadoop.metrics.log.= level=3DWARN 1638 -Djava.io.tmpdir=3D/tmp/hadoop-root/nm-local-dir/usercach= e/root/appcache/application_1426992254950_0002/container_1426992254950_0002= _01_000005/tmp -Dlog4j.configuration=3Dcontainer-log4j.properties -Dyarn.ap= p.container.log.dir=3D/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/applicati= on_1426992254950_0002/container_1426992254950_0002_01_000005 -Dyarn.app.con= tainer.log.filesize=3D0 -Dhadoop.root.logger=3DINFO,CLA org.apache.hadoop.m= apred.YarnChild 199.26.254.140 36542 attempt_1426992254950_0002_m_000003_0 = 5=C2=A0
|- 7954 7949 7954 7954 (bash) 0 0= 65421312 275 /bin/bash -c /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/j= re/bin/java -Djava.net.preferIPv4Stack=3Dtrue -Dhadoop.metrics.log.level=3D= WARN=C2=A0 1638 -Djava.io.tmpdir=3D/tmp/hadoop-root/nm-local-dir/usercache/= root/appcache/application_1426992254950_0002/container_1426992254950_0002_0= 1_000005/tmp -Dlog4j.configuration=3Dcontainer-log4j.properties -Dyarn.app.= container.log.dir=3D/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application= _1426992254950_0002/container_1426992254950_0002_01_000005 -Dyarn.app.conta= iner.log.filesize=3D0 -Dhadoop.root.logger=3DINFO,CLA org.apache.hadoop.map= red.YarnChild 199.26.254.140 36542 attempt_1426992254950_0002_m_000003_0 5 = 1>/home/hadoop-lzl/hadoop-2.6.0/logs/userlogs/application_1426992254950_= 0002/container_1426992254950_0002_01_000005/stdout 2>/home/hadoop-lzl/ha= doop-2.6.0/logs/userlogs/application_1426992254950_0002/container_142699225= 4950_0002_01_000005/stderr =C2=A0

Exception from container-launch.
Container id: contai= ner_1426992254950_0002_01_000005
Exit code: 1
Stack trace: ExitCodeException exitCode=3D1:=C2=A0<= /div>
at org.apache.hadoop.util.Shell.runComman= d(Shell.java:538)
at org.apache.hadoop.ut= il.Shell.run(Shell.java:455)
at org.apach= e.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at org.apache.hadoop.yarn.server.nodemanager.Defaul= tContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
=
at org.apache.hadoop.yarn.server.nodemanager.c= ontainermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager= .containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask$Sync.i= nnerRun(FutureTask.java:334)
at java.util= .concurrent.FutureTask.run(FutureTask.java:166)
= at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecu= tor.java:1110)
at java.util.concurrent.Th= readPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)

Thanks,
Fei

--047d7b4513d85e80b70511edadcf--