hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeffrey Zhong (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-8018) Add "Flaky Testcase Detector" tool into dev-tools
Date Sun, 11 Aug 2013 00:30:48 GMT

     [ https://issues.apache.org/jira/browse/HBASE-8018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jeffrey Zhong updated HBASE-8018:
---------------------------------

    Description: 
jenkins-tools
=============

A tool which pulls test case results from Jenkins server. It displays a union of failed test
cases 
from the last 15(by default and actual number of jobs can be less depending on availablity)
runs 
recorded in Jenkins sever and track how each of them are performed for all the last 15 runs(passed,

not run or failed)

*Pre-requirement(run under folder jenkins-tools)*
       Please download jenkins-client from https://github.com/cosmin/jenkins-client
       1) git clone git://github.com/cosmin/jenkins-client.git
       2) make sure the dependency jenkins-client version in ./buildstats/pom.xml matches
the 
          downloaded jenkins-client(current value is 0.1.6-SNAPSHOT)
       
Build command(run under folder jenkins-tools):
{code}
       mvn clean package
{code}
Usage are: 
{code}
       java -jar ./buildstats/target/buildstats.jar <Jenkins HTTP URL> <Job Name>
[number of last most recent jobs to check]
{code}
Sample commands are:
{code}
       java -jar ./buildstats/target/buildstats.jar https://builds.apache.org HBase-TRUNK
{code}
Sample output(where 1 means "PASSED", 0 means "NOT RUN AT ALL", -1 means "FAILED"):

Failed Test Cases Stats        4360 4361 4362 4363 4364 4365 4366 4367 4368 4369
========================================================
org.apache.hadoop.hbase.backup.testhfilearchiving.testcleaningrace    1    1    1    1   
1    1    1    1   -1    0
org.apache.hadoop.hbase.migration.testnamespaceupgrade.testrenameusingsnapshots    1    1
   1   -1    0    1    1    1    1    1

Skipped Test Cases Stats
======= 4360 skipped(Or don't have) following test suites =======
org.apache.hadoop.hbase.replication.testreplicationkillmasterrscompressed
org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfilessplitrecovery
org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfiles
org.apache.hadoop.hbase.mapreduce.testmapreduceexamples
======= 4361 skipped(Or don't have) following test suites =======
org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfilessplitrecovery
org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfiles
org.apache.hadoop.hbase.mapreduce.testmapreduceexamples
======= 4362 skipped(Or don't have) following test suites =======
org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfilessplitrecovery
org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfiles
org.apache.hadoop.hbase.mapreduce.testmapreduceexamples
======= 4363 skipped(Or don't have) following test suites =======
org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfilessplitrecovery
org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfiles
org.apache.hadoop.hbase.mapreduce.testmapreduceexamples
======= 4368 skipped(Or don't have) following test suites =======
org.apache.hadoop.hbase.client.testadmin
org.apache.hadoop.hbase.client.testclonesnapshotfromclient
org.apache.hadoop.hbase.mapreduce.testmapreduceexamples

  was:
Recently I'm working on some hbase test case failures and I think it's useful if we can see
a report of all failed test cases from most recent runs. So we can easily see how flaky a
test case is. I wrote a tool some time back and below are some reports against different branches
from today's run. You can get the tools' source at https://github.com/jeffreyz88/jenkins-tools.
If we can run it daily and send out an email then we can quickly notice what may break after
recent check-ins. 

Notes: 
1) " 873  874  875  876  877  878  879  880  881" are builds which has failed test cases in
current or previous runs
2) 1 means "PASSED", 0 means "NOT RUN AT ALL", -1 means "FAILED"

HBase-0.95 (from last 10 runs configurable)

Failed Test Cases                21   22   23   24   25   27
========================================================
org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb.org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb
   0    0    0    0   -1   -1
org.apache.hadoop.hbase.coprocessor.example.testbulkdeleteprotocol.testbulkdeletecolumn  
-1    0    0    0    0    0
org.apache.hadoop.hbase.coprocessor.example.testrowcountendpoint.org.apache.hadoop.hbase.coprocessor.example.testrowcountendpoint
  -1    0    0    0    0    0
org.apache.hadoop.hbase.coprocessor.example.testzookeeperscanpolicyobserver.org.apache.hadoop.hbase.coprocessor.example.testzookeeperscanpolicyobserver
  -1    0    0    0    0    0
org.apache.hadoop.hbase.master.testmasterfailover.testmasterfailoverwithmockedritondeadrs
   1   -1   -1    0    1    1
org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshouldthrowioexceptionifstorefilesizeisemptyandshouldsuccessfullyexecuterollback
   1    1    1    1    1   -1
org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshutdownfixupwhendaughterhassplit
   1    1    1   -1   -1   -1
org.apache.hadoop.hbase.regionserver.wal.testhlog.testlogcleaning    0    1    1    1   -1
   0
org.apache.hadoop.hbase.replication.testmasterreplication.testcyclicreplication    1    1
   1    1    1   -1
org.apache.hadoop.hbase.replication.testreplicationqueuefailover.queuefailover    1   -1 
  0    1   -1    0
org.apache.hadoop.hbase.replication.testreplicationqueuefailovercompressed.queuefailover 
  0    1   -1    0   -1    0
org.apache.hadoop.hbase.security.access.testaccesscontroller.org.apache.hadoop.hbase.security.access.testaccesscontroller
   0   -1    0    0    0    0

As you can see we have a few test cases don't run successfully at all or recently.

HBase-TRUNK (from last 10 runs)

Failed Test Cases              3908 3909 3910 3912 3913 3914 3915 3916
========================================================
org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb.org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb
   0   -1    0    0    0    0   -1   -1
org.apache.hadoop.hbase.client.testadmin.testcloseregionwhenservernameisempty    1    1  
 1    1    1    1   -1    0
org.apache.hadoop.hbase.client.testscannertimeout.test3686a    1    1   -1    0    1    1
   1    1
org.apache.hadoop.hbase.client.testsnapshotcloneindependence.testofflinesnapshotregionoperationsindependent
   0    1   -1    0    1    1    1    1
org.apache.hadoop.hbase.client.testsnapshotcloneindependence.testonlinesnapshotregionoperationsindependent
   1    1   -1    0    1    1    1    1
org.apache.hadoop.hbase.master.testassignmentmanageroncluster.testmoveregion    1    1   -1
   0    1    1    1    1
org.apache.hadoop.hbase.master.testdistributedlogsplitting.testdelayeddeleteonfailure    1
   1   -1    0    1    1    1    1
org.apache.hadoop.hbase.master.testmasterfailover.testmasterfailoverwithmockedritondeadrs
   1    1    1    1    1   -1    0    1
org.apache.hadoop.hbase.master.testtablelockmanager.testdelete    1    1   -1    0    1  
 1    1    1
org.apache.hadoop.hbase.procedure.testzkprocedurecontrollers.testsimplezkcohortmembercontroller
   1    1    1    1    1    1   -1    0
org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testexistingznodeblockssplitandwerollback
   1    1    1    1    1   -1    0    1
org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshouldthrowioexceptionifstorefilesizeisemptyandshouldsuccessfullyexecuterollback
   1    1   -1    0    1    1    1    1
org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshutdownfixupwhendaughterhassplit
  -1   -1   -1    0   -1    0   -1    0
org.apache.hadoop.hbase.regionserver.wal.testhlog.testlogcleaning    0   -1   -1    0    1
   1    1    1
org.apache.hadoop.hbase.replication.testmultislavereplication.testmultislavereplication  
 1    1   -1    0    1    1    1    1
org.apache.hadoop.hbase.replication.testreplicationqueuefailover.queuefailover    1    1 
  1   -1    0   -1    0    1
org.apache.hadoop.hbase.replication.testreplicationqueuefailovercompressed.queuefailover 
  1    1    1   -1    0   -1    0    1
org.apache.hadoop.hbase.security.access.testaccesscontrolfilter.testqualifieraccess   -1 
  0    1   -1    0    1    1    1
org.apache.hadoop.hbase.util.testhbasefsck.testdegenerateregions    1   -1    0    1    1
   1    1    1
org.apache.hadoop.hbase.util.testhbasefsck.testfixassignmentsandnohdfschecking    1   -1 
  0    1    1    1    1    1
org.apache.hadoop.hbase.util.testhbasefsck.testfixbytable    1   -1    0    1    1    1  
 1    1
org.apache.hadoop.hbase.util.testhbasefsck.testhbckthreadpooling    1   -1    0    1    1
   1    1    1
org.apache.hadoop.hbase.util.testhbasefsck.testlingeringreferencefile    1   -1    0    1
   1    1    1    1
org.apache.hadoop.hbase.util.testhbasefsck.testmissingfirstregion    1   -1    0    1    1
   1    1    1
org.apache.hadoop.hbase.util.testhbasefsck.testmissinglastregion    1   -1    0    1    1
   1    1    1
org.apache.hadoop.hbase.util.testhbasefsck.testnotinhdfs    1   -1    0    1    1    1   
1    1
org.apache.hadoop.hbase.util.testhbasefsck.testnotinmetaordeployedhole    1   -1    0    1
   1    1    1    1
org.apache.hadoop.hbase.util.testhbasefsck.testoverlapandorphan    1   -1    0    1    1 
  1    1    1
org.apache.hadoop.hbase.util.testhbasefsck.testregionhole    1   -1    0    1    1    1  
 1    1
org.apache.hadoop.hbase.util.testhbasefsck.testregionshouldnotbedeployed    1   -1    0  
 1    1    1    1    1
org.apache.hadoop.hbase.util.testhbasefsck.testsidelineoverlapregion    1   -1    0    1 
  1    1    1    1
org.apache.hadoop.hbase.util.testhbasefsck.testsplitdaughtersnotinmeta    1   -1    0    1
   1    1    1    1

More test cases are not in good shape.

HBase-0.94(from last 10 runs)

Failed Test Cases               873  874  875  876  877  878  879  880  881
========================================================
org.apache.hadoop.hbase.backup.testhfilearchiving.testcleaningrace   -1    0    1    1   
1    1    1    1    1
org.apache.hadoop.hbase.catalog.testmetareadereditor.testgetregion    1    1   -1    0   
1    1    1    1    1
org.apache.hadoop.hbase.catalog.testmetareadereditor.testretrying    1    1   -1    0    1
   1    1    1    1
org.apache.hadoop.hbase.catalog.testmetareadereditor.testscanmetafortable    1    1   -1 
  0    1    1    1    1    1
org.apache.hadoop.hbase.catalog.testmetareadereditor.testtableexists    1    1   -1    0 
  1    1    1    1    1
org.apache.hadoop.hbase.client.testmetamigrationremovinghtd.org.apache.hadoop.hbase.client.testmetamigrationremovinghtd
   0   -1    0    0    0    0    0    0    0
org.apache.hadoop.hbase.client.testmultiparallel.testflushcommitsnoabort    1    1    1  
 1    1   -1    0    1    1
org.apache.hadoop.hbase.client.testmultiparallel.testflushcommitswithabort    1    1    1
   1    1   -1    0    1    1
org.apache.hadoop.hbase.client.testshell.testrunshelltests    1    1    1    1    1    1 
 -1    0    1
org.apache.hadoop.hbase.master.testmasterfailover.testmasterfailoverwithmockedritondeadrs
   1    1   -1   -1    0    1    1    1    1
org.apache.hadoop.hbase.regionserver.testatomicoperation.testmultirowmutationmultithreads
   1    1    1   -1    0    1    1    1    1
org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshutdownfixupwhendaughterhassplit
   1   -1    0    1    1    1    1    1    1 

As you can see 0.94 is stable and testmetamigrationremovinghtd  hasn't been run for a while
either being removed or due to other failures. Some test cases aren't stable but not too bad.


Thanks,
-Jeffrey


    
> Add "Flaky Testcase Detector" tool into dev-tools
> -------------------------------------------------
>
>                 Key: HBASE-8018
>                 URL: https://issues.apache.org/jira/browse/HBASE-8018
>             Project: HBase
>          Issue Type: Bug
>          Components: util
>            Reporter: Jeffrey Zhong
>            Assignee: Jeffrey Zhong
>             Fix For: 0.98.0
>
>         Attachments: hbase-8018.patch, hbase-8018-v1.patch, hbase-8018-v2.patch
>
>
> jenkins-tools
> =============
> A tool which pulls test case results from Jenkins server. It displays a union of failed
test cases 
> from the last 15(by default and actual number of jobs can be less depending on availablity)
runs 
> recorded in Jenkins sever and track how each of them are performed for all the last 15
runs(passed, 
> not run or failed)
> *Pre-requirement(run under folder jenkins-tools)*
>        Please download jenkins-client from https://github.com/cosmin/jenkins-client
>        1) git clone git://github.com/cosmin/jenkins-client.git
>        2) make sure the dependency jenkins-client version in ./buildstats/pom.xml matches
the 
>           downloaded jenkins-client(current value is 0.1.6-SNAPSHOT)
>        
> Build command(run under folder jenkins-tools):
> {code}
>        mvn clean package
> {code}
> Usage are: 
> {code}
>        java -jar ./buildstats/target/buildstats.jar <Jenkins HTTP URL> <Job
Name> [number of last most recent jobs to check]
> {code}
> Sample commands are:
> {code}
>        java -jar ./buildstats/target/buildstats.jar https://builds.apache.org HBase-TRUNK
> {code}
> Sample output(where 1 means "PASSED", 0 means "NOT RUN AT ALL", -1 means "FAILED"):
> Failed Test Cases Stats        4360 4361 4362 4363 4364 4365 4366 4367 4368 4369
> ========================================================
> org.apache.hadoop.hbase.backup.testhfilearchiving.testcleaningrace    1    1    1   
1    1    1    1    1   -1    0
> org.apache.hadoop.hbase.migration.testnamespaceupgrade.testrenameusingsnapshots    1
   1    1   -1    0    1    1    1    1    1
> Skipped Test Cases Stats
> ======= 4360 skipped(Or don't have) following test suites =======
> org.apache.hadoop.hbase.replication.testreplicationkillmasterrscompressed
> org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfilessplitrecovery
> org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfiles
> org.apache.hadoop.hbase.mapreduce.testmapreduceexamples
> ======= 4361 skipped(Or don't have) following test suites =======
> org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfilessplitrecovery
> org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfiles
> org.apache.hadoop.hbase.mapreduce.testmapreduceexamples
> ======= 4362 skipped(Or don't have) following test suites =======
> org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfilessplitrecovery
> org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfiles
> org.apache.hadoop.hbase.mapreduce.testmapreduceexamples
> ======= 4363 skipped(Or don't have) following test suites =======
> org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfilessplitrecovery
> org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfiles
> org.apache.hadoop.hbase.mapreduce.testmapreduceexamples
> ======= 4368 skipped(Or don't have) following test suites =======
> org.apache.hadoop.hbase.client.testadmin
> org.apache.hadoop.hbase.client.testclonesnapshotfromclient
> org.apache.hadoop.hbase.mapreduce.testmapreduceexamples

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message