hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Hsieh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-8381) TestTableInputFormatScan on Hadoop 2 fails because YARN kills our applications
Date Fri, 19 Apr 2013 17:03:15 GMT

    [ https://issues.apache.org/jira/browse/HBASE-8381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13636602#comment-13636602
] 

Jonathan Hsieh commented on HBASE-8381:
---------------------------------------

I committed HBASE-6330 to 0.95 and turnk but not 0.94 and I believe it is related   (specifically,
the vmem setting in hadoop2 against hadoop 2.0.2-alpha).

{code}

diff --git hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
index 57e6acb..0a5406f 100644
--- hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
+++ hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
@@ -1648,6 +1648,10 @@ public class HBaseTestingUtility extends HBaseCommonTestingUtility
{
 
     forceChangeTaskLogDir();
 
+    // Tests were failing because this process used 6GB of virtual memory and was getting
killed.
+    // we up the VM usable so that processes don't get killed.
+    conf.setFloat("yarn.nodemanager.vmem-pmem-ratio", 8.0f);
+
     // Allow the user to override FS URI for this map-reduce cluster to use.
     mrCluster = new MiniMRCluster(servers,
       FS_URI != null ? FS_URI : FileSystem.get(conf).getUri().toString(), 1,
@@ -1656,6 +1660,7 @@ public class HBaseTestingUtility extends HBaseCommonTestingUtility {
     if (jobConf == null) {
       jobConf = mrCluster.createJobConf();
     }
+
     jobConf.set("mapred.local.dir",
       conf.get("mapred.local.dir")); //Hadoop MiniMR overwrites this while it should not
     LOG.info("Mini mapreduce cluster started");
@@ -1664,14 +1669,14 @@ public class HBaseTestingUtility extends HBaseCommonTestingUtility
{
     conf.set("mapred.job.tracker", jobConf.get("mapred.job.tracker"));
     // this for mrv2 support; mr1 ignores this
     conf.set("mapreduce.framework.name", "yarn");
-    String rmAdress = jobConf.get("yarn.resourcemanager.address");
-    if (rmAdress != null) {
-      conf.set("yarn.resourcemanager.address", rmAdress);
+    String rmAddress = jobConf.get("yarn.resourcemanager.address");
+    if (rmAddress != null) {
+      conf.set("yarn.resourcemanager.address", rmAddress);
     }
-    String schedulerAdress =
+    String schedulerAddress =
       jobConf.get("yarn.resourcemanager.scheduler.address");
-    if (schedulerAdress != null) {
-      conf.set("yarn.resourcemanager.scheduler.address", schedulerAdress);
+    if (schedulerAddress != null) {
+      conf.set("yarn.resourcemanager.scheduler.address", schedulerAddress);
     }
   }
{code}
                
> TestTableInputFormatScan on Hadoop 2 fails because YARN kills our applications
> ------------------------------------------------------------------------------
>
>                 Key: HBASE-8381
>                 URL: https://issues.apache.org/jira/browse/HBASE-8381
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.94.6
>            Reporter: Jean-Daniel Cryans
>
> Snippet from http://54.241.6.143/job/HBase-0.94-Hadoop-2/org.apache.hbase$hbase/72/testReport/junit/org.apache.hadoop.hbase.mapreduce/TestTableInputFormatScan1/testScanEmptyToEmpty/
> {noformat}
> 2013-04-16 04:12:06,097 WARN  [Container Monitor] monitor.ContainersMonitorImpl$MonitoringThread(436):
Container [pid=30006,containerID=container_1366085506903_0001_01_000001] is running beyond
virtual memory limits. Current usage: 160.9mb of 1.5gb physical memory used; 3.4gb of 3.1gb
virtual memory used. Killing container.
> Dump of the process-tree for container_1366085506903_0001_01_000001 :
> 	|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES)
RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> 	|- 30006 24535 30006 30006 (bash) 0 3 110800896 303 /bin/bash -c /usr/java/default/bin/java
-Dlog4j.configuration=container-log4j.properties -Dyarn.app.mapreduce.container.log.dir=/home/ec2-user/jenkins/workspace/HBase-0.94-Hadoop-2/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-1_1/application_1366085506903_0001/container_1366085506903_0001_01_000001
-Dyarn.app.mapreduce.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Xmx1024m org.apache.hadoop.mapreduce.v2.app.MRAppMaster
1>/home/ec2-user/jenkins/workspace/HBase-0.94-Hadoop-2/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-1_1/application_1366085506903_0001/container_1366085506903_0001_01_000001/stdout
2>/home/ec2-user/jenkins/workspace/HBase-0.94-Hadoop-2/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-1_1/application_1366085506903_0001/container_1366085506903_0001_01_000001/stderr
  
> 	|- 30024 30006 30006 30006 (java) 714 62 3517714432 40881 /usr/java/default/bin/java
-Dlog4j.configuration=container-log4j.properties -Dyarn.app.mapreduce.container.log.dir=/home/ec2-user/jenkins/workspace/HBase-0.94-Hadoop-2/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-1_1/application_1366085506903_0001/container_1366085506903_0001_01_000001
-Dyarn.app.mapreduce.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Xmx1024m org.apache.hadoop.mapreduce.v2.app.MRAppMaster

> {noformat}
> It started with this build according to my gmail search: http://54.241.6.143/job/HBase-0.94-Hadoop-2/70/
> Related to HBASE-8326 somehow, [~ndimiduk]?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message