hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinod Kumar Vavilapalli <vino...@hortonworks.com>
Subject Re: Container [pid=22885,containerID=container_1386156666044_0001_01_000013] is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical memory used; 332.5 GB of 8 GB virtual memory used. Killing container.
Date Fri, 06 Dec 2013 03:32:33 GMT
Something looks really bad on your cluster. The JVM's heap size is 200MB
but its virtual memory has ballooned to a monstrous 332GB. Does that ring
any bell? Can you run regular java applications on this node? This doesn't
seem related to YARN per-se.

+Vinod
Hortonworks Inc.
http://hortonworks.com/


On Wed, Dec 4, 2013 at 5:16 AM, panfei <cnweike@gmail.com> wrote:

>
>
> ---------- Forwarded message ----------
> From: panfei <cnweike@gmail.com>
> Date: 2013/12/4
> Subject: Container
> [pid=22885,containerID=container_1386156666044_0001_01_000013] is running
> beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical
> memory used; 332.5 GB of 8 GB virtual memory used. Killing container.
> To: CDH Users <cdh-user@cloudera.org>
>
>
> Hi All:
>
> We are using CDH4.5 Hadoop for our production, when submit some (not all)
> jobs from hive, we get the following exception info , seems the physical
> memory and virtual memory both not enough for the job to run:
>
>
> Task with the most failures(4):
> -----
> Task ID:
>   task_1386156666044_0001_m_000000
>
> URL:
>
> http://namenode-1:8088/taskdetails.jsp?jobid=job_1386156666044_0001&tipid=task_1386156666044_0001_m_000000
> -----
> Diagnostic Messages for this Task:
> Container [pid=22885,containerID=container_1386156666044_0001_01_000013]
> is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB
> physical memory used; 332.5 GB of 8 GB virtual memory used. Killing
> container.
> Dump of the process-tree for container_1386156666044_0001_01_000013 :
>         |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>         |- 22885 22036 22885 22885 (java) 5414 108 356993519616 271953
> /usr/java/default/bin/java -Djava.net.preferIPv4Stack=true
> -Dhadoop.metrics.log.level=WARN -Xmx200m
> -Djava.io.tmpdir=/data/yarn/local/usercache/hive/appcache/application_1386156666044_0001/container_1386156666044_0001_01_000013/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.mapreduce.container.log.dir=/var/log/hadoop-yarn/containers/application_1386156666044_0001/container_1386156666044_0001_01_000013
> -Dyarn.app.mapreduce.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
> org.apache.hadoop.mapred.YarnChild 192.168.101.55 60841
> attempt_1386156666044_0001_m_000000_3 13
>
> following is some of our configuration:
>
>   <property>
>     <name>yarn.nodemanager.resource.memory-mb</name>
>     <value>12288</value>
>   </property>
>
>   <property>
>     <name>yarn.nodemanager.vmem-pmem-ratio</name>
>     <value>8</value>
>   </property>
>
>   <property>
>     <name>yarn.nodemanager.vmem-check-enabled</name>
>     <value>false</value>
>   </property>
>
>   <property>
>     <name>yarn.nodemanager.resource.cpu-vcores</name>
>     <value>6</value>
>   </property>
>
> can you give me some advice? thanks a lot.
> --
> 不学习,不知道
>
>
>
> --
> 不学习,不知道
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
View raw message