hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: Flaky Testcase Detector
Date Wed, 06 Mar 2013 03:12:30 GMT
This is handy. Thanks so much for sharing it.

On Wednesday, March 6, 2013, Jeffrey Zhong wrote:

> Hey,
>
> Recently I'm working on some hbase test case failures and I think it's
> useful if we can see a report of all failed test cases from most recent
> runs. So we can easily see how flaky a test case is. I wrote a tool some
> time back and below are some reports against different branches from
> today's
> run. You can get the tools' source at
> https://github.com/jeffreyz88/jenkins-tools. If we can run it daily and
> send
> out an email then we can quickly notice what may break after recent
> check-ins.
>
> Notes:
> 1) " 873  874  875  876  877  878  879  880  881" are builds which has
> failed test cases in current or previous runs
> 2) 1 means "PASSED", 0 means "NOT RUN AT ALL", -1 means "FAILED"
>
> HBase-0.95 (from last 10 runs configurable)
>
> Failed Test Cases                21   22   23   24   25   27
> ========================================================
>
> org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb.org.apache.h
> adoop.hbase.catalog.testmetamigrationconvertingtopb    0    0    0    0
> -1
> -1
>
> org.apache.hadoop.hbase.coprocessor.example.testbulkdeleteprotocol.testbulkd
> eletecolumn   -1    0    0    0    0    0
>
> org.apache.hadoop.hbase.coprocessor.example.testrowcountendpoint.org.apache.
> hadoop.hbase.coprocessor.example.testrowcountendpoint   -1    0    0    0
> 0    0
>
> org.apache.hadoop.hbase.coprocessor.example.testzookeeperscanpolicyobserver.
> org.apache.hadoop.hbase.coprocessor.example.testzookeeperscanpolicyobserver
> -1    0    0    0    0    0
>
> org.apache.hadoop.hbase.master.testmasterfailover.testmasterfailoverwithmock
> edritondeadrs    1   -1   -1    0    1    1
>
> org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshoul
> dthrowioexceptionifstorefilesizeisemptyandshouldsuccessfullyexecuterollback
> 1    1    1    1    1   -1
>
> org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshutd
> ownfixupwhendaughterhassplit    1    1    1   -1   -1   -1
> org.apache.hadoop.hbase.regionserver.wal.testhlog.testlogcleaning    0    1
> 1    1   -1    0
>
> org.apache.hadoop.hbase.replication.testmasterreplication.testcyclicreplicat
> ion    1    1    1    1    1   -1
>
> org.apache.hadoop.hbase.replication.testreplicationqueuefailover.queuefailov
> er    1   -1    0    1   -1    0
>
> org.apache.hadoop.hbase.replication.testreplicationqueuefailovercompressed.q
> ueuefailover    0    1   -1    0   -1    0
>
> org.apache.hadoop.hbase.security.access.testaccesscontroller.org.apache.hado
> op.hbase.security.access.testaccesscontroller    0   -1    0    0    0    0
>
> As you can see we have a few test cases don't run successfully at all or
> recently.
>
> HBase-TRUNK (from last 10 runs)
>
> Failed Test Cases              3908 3909 3910 3912 3913 3914 3915 3916
> ========================================================
>
> org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb.org.apache.h
> adoop.hbase.catalog.testmetamigrationconvertingtopb    0   -1    0    0
>  0
> 0   -1   -1
>
> org.apache.hadoop.hbase.client.testadmin.testcloseregionwhenservernameisempt
> y    1    1    1    1    1    1   -1    0
> org.apache.hadoop.hbase.client.testscannertimeout.test3686a    1    1   -1
> 0    1    1    1    1
>
> org.apache.hadoop.hbase.client.testsnapshotcloneindependence.testofflinesnap
> shotregionoperationsindependent    0    1   -1    0    1    1    1    1
>
> org.apache.hadoop.hbase.client.testsnapshotcloneindependence.testonlinesnaps
> hotregionoperationsindependent    1    1   -1    0    1    1    1    1
>
> org.apache.hadoop.hbase.master.testassignmentmanageroncluster.testmoveregion
> 1    1   -1    0    1    1    1    1
>
> org.apache.hadoop.hbase.master.testdistributedlogsplitting.testdelayeddelete
> onfailure    1    1   -1    0    1    1    1    1
>
> org.apache.hadoop.hbase.master.testmasterfailover.testmasterfailoverwithmock
> edritondeadrs    1    1    1    1    1   -1    0    1
> org.apache.hadoop.hbase.master.testtablelockmanager.testdelete    1    1
> -1    0    1    1    1    1
>
> org.apache.hadoop.hbase.procedure.testzkprocedurecontrollers.testsimplezkcoh
> ortmembercontroller    1    1    1    1    1    1   -1    0
>
> org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testexist
> ingznodeblockssplitandwerollback    1    1    1    1    1   -1    0    1
>
> org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshoul
> dthrowioexceptionifstorefilesizeisemptyandshouldsuccessfullyexecuterollback
> 1    1   -1    0    1    1    1    1
>
> org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshutd
> ownfixupwhendaughterhassplit   -1   -1   -1    0   -1    0   -1    0
> org.apache.hadoop.hbase.regionserver.wal.testhlog.testlogcleaning    0   -1
> -1    0    1    1    1    1
>
> org.apache.hadoop.hbase.replication.testmultislavereplication.testmultislave
> replication    1    1   -1    0    1    1    1    1
>
> org.apache.hadoop.hbase.replication.testreplicationqueuefailover.queuefailov
> er    1    1    1   -1    0   -1    0    1
>
> org.apache.hadoop.hbase.replication.testreplicationqueuefailovercompressed.q
> ueuefailover    1    1    1   -1    0   -1    0    1
>
> org.apache.hadoop.hbase.security.access.testaccesscontrolfilter.testqualifie
> raccess   -1    0    1   -1    0    1    1    1
> org.apache.hadoop.hbase.util.testhbasefsck.testdegenerateregions    1   -1
> 0    1    1    1    1    1
>
> org.apache.hadoop.hbase.util.testhbasefsck.testfixassignmentsandnohdfschecki
> ng    1   -1    0    1    1    1    1    1
> org.apache.hadoop.hbase.util.testhbasefsck.testfixbytable    1   -1    0
> 1    1    1    1    1
> org.apache.hadoop.hbase.util.testhbasefsck.testhbckthreadpooling    1   -1
> 0    1    1    1    1    1
> org.apache.hadoop.hbase.util.testhbasefsck.testlingeringreferencefile    1
> -1    0    1    1    1    1    1
> org.apache.hadoop.hbase.util.testhbasefsck.testmissingfirstregion    1   -1
> 0    1    1    1    1    1
> org.apache.hadoop.hbase.util.testhbasefsck.testmissinglastregion    1   -1
> 0    1    1    1    1    1
> org.apache.hadoop.hbase.util.testhbasefsck.testnotinhdfs    1   -1    0
>  1
> 1    1    1    1
> org.apache.hadoop.hbase.util.testhbasefsck.testnotinmetaordeployedhole    1
> -1    0    1    1    1    1    1
> org.apache.hadoop.hbase.util.testhbasefsck.testoverlapandorphan    1   -1
> 0    1    1    1    1    1
> org.apache.hadoop.hbase.util.testhbasefsck.testregionhole    1   -1    0
> 1    1    1    1    1
> org.apache.hadoop.hbase.util.testhbasefsck.testregionshouldnotbedeployed
> 1   -1    0    1    1    1    1    1
> org.apache.hadoop.hbase.util.testhbasefsck.testsidelineoverlapregion    1
> -1    0    1    1    1    1    1
> org.apache.hadoop.hbase.util.testhbasefsck.testsplitdaughtersnotinmeta    1
> -1    0    1    1    1    1    1
>
> More test cases are not in good shape.
>
> HBase-0.94(from last 10 runs)
>
> Failed Test Cases               873  874  875  876  877  878  879  880  881
> ========================================================
> org.apache.hadoop.hbase.backup.testhfilearchiving.testcleaningrace   -1
>  0
> 1    1    1    1    1    1    1
> org.apache.hadoop.hbase.catalog.testmetareadereditor.testgetregion    1
>  1
> -1    0    1    1    1    1    1
> org.apache.hadoop.hbase.catalog.testmetareadereditor.testretrying    1    1
> -1    0    1    1    1    1    1
> org.apache.hadoop.hbase.catalog.testmetareadereditor.testscanmetafortable
> 1    1   -1    0    1    1    1    1    1
> org.apache.hadoop.hbase.catalog.testmetareadereditor.testtableexists    1
> 1   -1    0    1    1    1    1    1
>
> org.apache.hadoop.hbase.client.testmetamigrationremovinghtd.org.apache.hadoo
> p.hbase.client.testmetamigrationremovinghtd    0   -1    0    0    0    0
> 0    0    0
> org.apache.hadoop.hbase.client.testmultiparallel.testflushcommitsnoabort
> 1    1    1    1    1   -1    0    1    1
> org.apache.hadoop.hbase.client.testmultiparallel.testflushcommitswithabort
> 1    1    1    1    1   -1    0    1    1
> org.apache.hadoop.hbase.client.testshell.testrunshelltests    1    1    1
> 1    1    1   -1    0    1
>
> org.apache.hadoop.hbase.master.testmasterfailover.testmasterfailoverwithmock
> edritondeadrs    1    1   -1   -1    0    1    1    1    1
>
> org.apache.hadoop.hbase.regionserver.testatomicoperation.testmultirowmutatio
> nmultithreads    1    1    1   -1    0    1    1    1    1
>
> org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshutd
> ownfixupwhendaughterhassplit    1   -1    0    1    1    1    1    1    1
>
> As you can see 0.94 is stable and testmetamigrationremovinghtd  hasn't been
> run for a while either being removed or due to other failures. Some test
> cases aren't stable but not too bad.
>
> Thanks,
> -Jeffrey
>
>
>
>

-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message