hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "JR Wang" <jerrywang1...@163.com>
Subject Re:Re: Yarn container use huge virtual memory with JVM option -XX:+PrintGCDetails added.
Date Wed, 04 May 2016 08:58:37 GMT
Hi,


I did enabled the vmem check. But what I really want to know is that why huge virtual memory(17.9
GB) used when JVM option -XX:+PrintGCDetails was added.


The appliction which I run was hadoop jar hadoop-mapreduce-examples-2.7.2.jar pi -D mapreduce.map.java.opts=-XX:+PrintGCDetails
5 100.


If I remove the JVM options -XX:+PrintGCDetails, the application complete normally.


Thanks
JR




在2016年05月01 16时19分, "Varun Vasudev"<vvasudev@apache.org>写道:


Hello!


From the attached yarn-site.xml it looks like the vmem check is enabled. The value should
be set to false and the nodemanager restarted.


   <property>        <name>yarn.nodemanager.vmem-check-enabled</name>  
     <value>true</value>    </property>


-Varun


From: JR Wang <jerrywang1992@163.com>
Date: Friday, April 29, 2016 at 7:39 AM
To: user <user@hadoop.apache.org>
Subject: Yarn container use huge virtual memory with JVM option -XX:+PrintGCDetails added.




Hi All,

I’m currently running Hadoop 2.7.2 on my three node cluster, all node equiped with 32-core
CPU and 64 GB memory, Ubuntu 14.04.3 LTS. Yarn’s configuration kept the same with default
one, except yarn.nodemanager.resource.memory-mb set as 16384(16G).

Everything worked fine, but when I try to run Map/Reduce task with JVM options mapreduce.map.java.opts=-XX:+PrintGCDetails,
the task failed, with error message said that container is running beyond virtual memory limits.
Current usage: 212.0 MB of 1 GB physical memory used; 17.9 GB of 2.1 GB virtual memory used.
Killing container.

Actually I was just running the Map/Reduce expample pi with tiny calculation, hadoop jar hadoop-mapreduce-examples-2.7.2.jar
pi -D mapreduce.map.java.opts=-XX:+PrintGCDetails 5 100.

I’ve tried to disable yarn.nodemanager.vmem-check-enabled, but as calculation increasees,
error happens again.

I am really confused why huge virtual memory allcated when JVM option -XX:+PrintGCDetails
was added. Please help!

Thank You,
JR

Error Message

16/04/29 09:43:47 INFO mapreduce.Job: Task Id : attempt_1461846504106_0012_m_000004_1, Status
: FAILED
Container [pid=16629,containerID=container_1461846504106_0012_01_000010] is running beyond
virtual memory limits. Current usage: 212.0 MB of 1 GB physical memory used; 17.9 GB of 2.1
GB virtual memory used. Killing container.
Dump of the process-tree for container_1461846504106_0012_01_000010 :
  |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES)
RSSMEM_USAGE(PAGES) FULL_CMD_LINE
  |- 16629 16627 16629 16629 (bash) 0 0 17051648 667 /bin/bash -c /usr/lib/jvm/jdk8/bin/java
-Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -XX:+PrintGCDetails -Djava.io.tmpdir=/opt/hadoop-2.7.2/tmp/nodemanager/local/usercache/hadoop/appcache/application_1461846504106_0012/container_1461846504106_0012_01_000010/tmp
-Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/hadoop-2.7.2/tmp/nodemanager/logs/application_1461846504106_0012/container_1461846504106_0012_01_000010
-Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog
org.apache.hadoop.mapred.YarnChild 10.13.71.82 56755 attempt_1461846504106_0012_m_000004_1
10 1>/opt/hadoop-2.7.2/tmp/nodemanager/logs/application_1461846504106_0012/container_1461846504106_0012_01_000010/stdout
2>/opt/hadoop-2.7.2/tmp/nodemanager/logs/application_1461846504106_0012/container_1461846504106_0012_01_000010/stderr
 
  |- 16633 16629 16629 16629 (java) 399 25 19239059456 53616 /usr/lib/jvm/jdk8/bin/java -Djava.net.preferIPv4Stack=true
-Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -Djava.io.tmpdir=/opt/hadoop-2.7.2/tmp/nodemanager/local/usercache/hadoop/appcache/application_1461846504106_0012/container_1461846504106_0012_01_000010/tmp
-Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/hadoop-2.7.2/tmp/nodemanager/logs/application_1461846504106_0012/container_1461846504106_0012_01_000010
-Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog
org.apache.hadoop.mapred.YarnChild 10.13.71.82 56755 attempt_1461846504106_0012_m_000004_1
10

yarn-site.xml

<configuration><!-- Site specific YARN configuration properties -->
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>namenode:8032</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>namenode:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>namenode:8031</value>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>namenode:8033</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>namenode:8088</value>
    </property>
    <property>
        <name>yarn.nodemanager.local-dirs</name>
        <value>${hadoop.tmp.dir}/nodemanager/local</value>
    </property>
    <property>
        <name>yarn.nodemanager.log-dirs</name>
        <value>${hadoop.tmp.dir}/nodemanager/logs</value>
    </property>  
    <property>
        <name>yarn.nodemanager.log.retain-seconds</name>
        <value>10800</value>
    </property>
    <property>
        <!-- Where to aggregate logs to. -->
        <name>yarn.nodemanager.remote-app-log-dir</name>
        <value>${hadoop.tmp.dir}/nodemanager/remote</value>
    </property>
    <property>
        <name>yarn.nodemanager.remote-app-log-dir-suffix</name>
        <value>logs</value>
    </property>
    <property>
        <!-- enable log aggregation -->
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
    </property>
    <property>
        <!-- How long to keep aggregation logs before deleting them. -1 disables. -->
        <name>yarn.log-aggregation.retain-seconds</name>
        <!-- 30 days -->
        <value>2592000</value>
    </property>
    <property>
        <!-- How long to wait between aggregated log retention checks. If set to 0 or a
negative value then the value is computed as one-tenth of the aggregated log retention time.
 -->
        <name>yarn.log-aggregation.retain-check-interval-seconds</name>
        <value>-1</value>
    </property>
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>16384</value>
    </property>
    <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>true</value>
    </property></configuration>
Mime
View raw message