hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From panfei <cnwe...@gmail.com>
Subject Re: Container [pid=22885,containerID=container_1386156666044_0001_01_000013] is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical memory used; 332.5 GB of 8 GB virtual memory used. Killing container.
Date Mon, 09 Dec 2013 10:04:09 GMT
Hi All, thanks for your replies, I have found the point: there is a Memory
Leak(open a file for each record) in the Hive UDF method, after fixing it,
everything goes well now.


2013/12/6 Vinod Kumar Vavilapalli <vinodkv@hortonworks.com>

> Something looks really bad on your cluster. The JVM's heap size is 200MB
> but its virtual memory has ballooned to a monstrous 332GB. Does that ring
> any bell? Can you run regular java applications on this node? This doesn't
> seem related to YARN per-se.
>
> +Vinod
> Hortonworks Inc.
> http://hortonworks.com/
>
>
> On Wed, Dec 4, 2013 at 5:16 AM, panfei <cnweike@gmail.com> wrote:
>
>>
>>
>> ---------- Forwarded message ----------
>> From: panfei <cnweike@gmail.com>
>> Date: 2013/12/4
>> Subject: Container
>> [pid=22885,containerID=container_1386156666044_0001_01_000013] is running
>> beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical
>> memory used; 332.5 GB of 8 GB virtual memory used. Killing container.
>> To: CDH Users <cdh-user@cloudera.org>
>>
>>
>> Hi All:
>>
>> We are using CDH4.5 Hadoop for our production, when submit some (not all)
>> jobs from hive, we get the following exception info , seems the physical
>> memory and virtual memory both not enough for the job to run:
>>
>>
>> Task with the most failures(4):
>> -----
>> Task ID:
>>   task_1386156666044_0001_m_000000
>>
>> URL:
>>
>> http://namenode-1:8088/taskdetails.jsp?jobid=job_1386156666044_0001&tipid=task_1386156666044_0001_m_000000
>> -----
>> Diagnostic Messages for this Task:
>> Container [pid=22885,containerID=container_1386156666044_0001_01_000013]
>> is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB
>> physical memory used; 332.5 GB of 8 GB virtual memory used. Killing
>> container.
>> Dump of the process-tree for container_1386156666044_0001_01_000013 :
>>         |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
>> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>>         |- 22885 22036 22885 22885 (java) 5414 108 356993519616 271953
>> /usr/java/default/bin/java -Djava.net.preferIPv4Stack=true
>> -Dhadoop.metrics.log.level=WARN -Xmx200m
>> -Djava.io.tmpdir=/data/yarn/local/usercache/hive/appcache/application_1386156666044_0001/container_1386156666044_0001_01_000013/tmp
>> -Dlog4j.configuration=container-log4j.properties
>> -Dyarn.app.mapreduce.container.log.dir=/var/log/hadoop-yarn/containers/application_1386156666044_0001/container_1386156666044_0001_01_000013
>> -Dyarn.app.mapreduce.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
>> org.apache.hadoop.mapred.YarnChild 192.168.101.55 60841
>> attempt_1386156666044_0001_m_000000_3 13
>>
>> following is some of our configuration:
>>
>>   <property>
>>     <name>yarn.nodemanager.resource.memory-mb</name>
>>     <value>12288</value>
>>   </property>
>>
>>   <property>
>>     <name>yarn.nodemanager.vmem-pmem-ratio</name>
>>     <value>8</value>
>>   </property>
>>
>>   <property>
>>     <name>yarn.nodemanager.vmem-check-enabled</name>
>>     <value>false</value>
>>   </property>
>>
>>   <property>
>>     <name>yarn.nodemanager.resource.cpu-vcores</name>
>>     <value>6</value>
>>   </property>
>>
>> can you give me some advice? thanks a lot.
>> --
>> 不学习,不知道
>>
>>
>>
>> --
>> 不学习,不知道
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.




-- 
不学习,不知道

Mime
View raw message