hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Hsieh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-7605) TestMiniClusterLoadSequential fails in trunk build on hadoop 2
Date Wed, 10 Apr 2013 17:12:19 GMT

    [ https://issues.apache.org/jira/browse/HBASE-7605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13627999#comment-13627999
] 

Jonathan Hsieh commented on HBASE-7605:
---------------------------------------

tl;dr Test is correct but there is some regression when using hadoop2.  Going to reduce the
size of the test so that it finishes in a reasonable amount of time and passes.  Would like
to close with patch that makes it work but will file jira to investigate why the big h1 vs
h2 perf difference.

On my test runs against hadoop2, I get failures due to timeouts with TestMiniClusterLoadParallel
and TestMiniClusterLoadSequential.

With hadoop1 runs from the ec2 test trunk on hadoop2 instance [1], each of these individual
cases take this long to run.

*Sequential
{code}
Test name       Duration   Status   
loadTest[0] 	1 min 11 sec	Passed
loadTest[1] 	48 sec	Passed
loadTest[2] 	22 sec	Passed
loadTest[3] 	27 sec	Passed
{code}

*Parallel 
{code}
Test name   Duration   Status   
loadTest[0] 	1 min 18 sec	Passed
loadTest[1] 	51 sec	Passed
loadTest[2] 	19 sec	Passed
loadTest[3] 	20 sec	Passed
{code}

I ran then locally on my box with hadoop1 profile an got this:


With hadoop2 I did some experiments on my local machine, bumping up Sequential from 180s to
360s timeouts per test.   They passed, but barely before timing out.  For some reason they
are 2-3x slower.  (Not good)

{code}
  <testcase time="348.761" classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel"
name="loadTest[0]"/>
  <testcase time="343.577" classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel"
name="loadTest[1]"/>
  <testcase time="46.397" classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel"
name="loadTest[2]"/>
  <testcase time="45.845" classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel"
name="loadTest[3]"/>
...
  <testcase time="358.088" classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential"
name="loadTest[0]"/>
  <testcase time="357.654" classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential"
name="loadTest[1]"/>
  <testcase time="61.306" classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential"
name="loadTest[2]"/>
  <testcase time="61.263" classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential"
name="loadTest[3]"/>
{code}

Instead of bumping timeout up, I'm going to change the tests so that it does an order of magnitude
less work, and finishes on the order of 10-30s instead of 100-300s.  

Here are the hadoop2 vs hadoop1 results (bumping down from 10000 keys to 1000 keys)
{code}
hadoop1:
  <testcase time="14.929" classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel"
name="loadTest[0]"/>
  <testcase time="12.747" classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel"
name="loadTest[1]"/>
  <testcase time="12.005" classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel"
name="loadTest[2]"/>
  <testcase time="11.663" classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel"
name="loadTest[3]"/>

  <testcase time="15.956" classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential"
name="loadTest[0]"/>
  <testcase time="14.321" classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential"
name="loadTest[1]"/>
  <testcase time="11.541" classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential"
name="loadTest[2]"/>
  <testcase time="11.588" classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential"
name="loadTest[3]"/>

hadoop2:
  <testcase time="43.703" classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel"
name="loadTest[0]"/>
  <testcase time="41.542" classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel"
name="loadTest[1]"/>
  <testcase time="10.711" classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel"
name="loadTest[2]"/>
  <testcase time="8.801" classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel"
name="loadTest[3]"/>
  
  <testcase time="45.986" classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential"
name="loadTest[0]"/>
  <testcase time="42.037" classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential"
name="loadTest[1]"/>
  <testcase time="12.56" classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential"
name="loadTest[2]"/>
  <testcase time="12.01" classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential"
name="loadTest[3]"/>
{code}


[1]http://54.241.6.143/job/HBase-TRUNK-Hadoop-2/org.apache.hbase$hbase-server/53/
                
> TestMiniClusterLoadSequential fails in trunk build on hadoop 2
> --------------------------------------------------------------
>
>                 Key: HBASE-7605
>                 URL: https://issues.apache.org/jira/browse/HBASE-7605
>             Project: HBase
>          Issue Type: Sub-task
>            Reporter: Ted Yu
>            Priority: Critical
>             Fix For: 0.95.1
>
>
> From HBase-TRUNK-on-Hadoop-2.0.0 #354:
>   loadTest[0](org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential): test timed
out after 120000 milliseconds
>   loadTest[1](org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential): test timed
out after 120000 milliseconds
>   loadTest[2](org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential): test timed
out after 120000 milliseconds
>   loadTest[3](org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential): test timed
out after 120000 milliseconds

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message