Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates
 209.85.214.50 as permitted sender)
MIME-Version: 1.0
Sender: saint.ack@gmail.com
In-Reply-To: <CD5CC3CA.330E%jzhong@hortonworks.com>
References: 
 <CADcMMgESuh+u4Kk0fz-2JoACoFwOF7UE1LN08V=aEQmDsGNHBw@mail.gmail.com>
	<CD5CC3CA.330E%jzhong@hortonworks.com>
Date: Wed, 31 Jul 2013 16:43:52 -0700
Message-ID: 
 <CADcMMgGj71kKnn1c5hvjhoi3Oq_EftiCaLW_1RoYba0wjY9_YQ@mail.gmail.com>
Subject: Re: Flaky Testcase Detector
From: Stack <stack@duboce.net>
To: HBase Dev List <dev@hbase.apache.org>
Content-Type: multipart/alternative; boundary=001a11c36e004c2b8404e2d74c20

--001a11c36e004c2b8404e2d74c20
Content-Type: text/plain; charset=UTF-8

Did this get checked in?
Thanks,
St.Ack


On Wed, Mar 6, 2013 at 10:11 AM, Jeffrey Zhong <jzhong@hortonworks.com>wrote:

>
> I can check it into ./dev-tools unless someone objects.
>
> Thanks,
> -Jeffrey
>
> On 3/5/13 10:43 PM, "Stack" <stack@duboce.net> wrote:
>
> >Nice.  Should we check it into ./dev-tools?
> >Thanks,
> >St.Ack
> >
> >
> >On Tue, Mar 5, 2013 at 7:12 PM, Andrew Purtell <apurtell@apache.org>
> >wrote:
> >
> >> This is handy. Thanks so much for sharing it.
> >>
> >> On Wednesday, March 6, 2013, Jeffrey Zhong wrote:
> >>
> >> > Hey,
> >> >
> >> > Recently I'm working on some hbase test case failures and I think it's
> >> > useful if we can see a report of all failed test cases from most
> >>recent
> >> > runs. So we can easily see how flaky a test case is. I wrote a tool
> >>some
> >> > time back and below are some reports against different branches from
> >> > today's
> >> > run. You can get the tools' source at
> >> > https://github.com/jeffreyz88/jenkins-tools. If we can run it daily
> >>and
> >> > send
> >> > out an email then we can quickly notice what may break after recent
> >> > check-ins.
> >> >
> >> > Notes:
> >> > 1) " 873  874  875  876  877  878  879  880  881" are builds which has
> >> > failed test cases in current or previous runs
> >> > 2) 1 means "PASSED", 0 means "NOT RUN AT ALL", -1 means "FAILED"
> >> >
> >> > HBase-0.95 (from last 10 runs configurable)
> >> >
> >> > Failed Test Cases                21   22   23   24   25   27
> >> > ========================================================
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb.org.apach
> >>e.h
> >> > adoop.hbase.catalog.testmetamigrationconvertingtopb    0    0    0
> >>0
> >> > -1
> >> > -1
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.coprocessor.example.testbulkdeleteprotocol.testbu
> >>lkd
> >> > eletecolumn   -1    0    0    0    0    0
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.coprocessor.example.testrowcountendpoint.org.apac
> >>he.
> >> > hadoop.hbase.coprocessor.example.testrowcountendpoint   -1    0    0
> >>  0
> >> > 0    0
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.coprocessor.example.testzookeeperscanpolicyobserv
> >>er.
> >> >
> >>
> >>org.apache.hadoop.hbase.coprocessor.example.testzookeeperscanpolicyobserv
> >>er
> >> > -1    0    0    0    0    0
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.master.testmasterfailover.testmasterfailoverwithm
> >>ock
> >> > edritondeadrs    1   -1   -1    0    1    1
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testsh
> >>oul
> >> >
> >>
> >>dthrowioexceptionifstorefilesizeisemptyandshouldsuccessfullyexecuterollba
> >>ck
> >> > 1    1    1    1    1   -1
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testsh
> >>utd
> >> > ownfixupwhendaughterhassplit    1    1    1   -1   -1   -1
> >> > org.apache.hadoop.hbase.regionserver.wal.testhlog.testlogcleaning    0
> >>  1
> >> > 1    1   -1    0
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.replication.testmasterreplication.testcyclicrepli
> >>cat
> >> > ion    1    1    1    1    1   -1
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.replication.testreplicationqueuefailover.queuefai
> >>lov
> >> > er    1   -1    0    1   -1    0
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.replication.testreplicationqueuefailovercompresse
> >>d.q
> >> > ueuefailover    0    1   -1    0   -1    0
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.security.access.testaccesscontroller.org.apache.h
> >>ado
> >> > op.hbase.security.access.testaccesscontroller    0   -1    0    0    0
> >>  0
> >> >
> >> > As you can see we have a few test cases don't run successfully at all
> >>or
> >> > recently.
> >> >
> >> > HBase-TRUNK (from last 10 runs)
> >> >
> >> > Failed Test Cases              3908 3909 3910 3912 3913 3914 3915 3916
> >> > ========================================================
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb.org.apach
> >>e.h
> >> > adoop.hbase.catalog.testmetamigrationconvertingtopb    0   -1    0
> >>0
> >> >  0
> >> > 0   -1   -1
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.client.testadmin.testcloseregionwhenservernameise
> >>mpt
> >> > y    1    1    1    1    1    1   -1    0
> >> > org.apache.hadoop.hbase.client.testscannertimeout.test3686a    1    1
> >> -1
> >> > 0    1    1    1    1
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.client.testsnapshotcloneindependence.testofflines
> >>nap
> >> > shotregionoperationsindependent    0    1   -1    0    1    1    1
> >>1
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.client.testsnapshotcloneindependence.testonlinesn
> >>aps
> >> > hotregionoperationsindependent    1    1   -1    0    1    1    1    1
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.master.testassignmentmanageroncluster.testmovereg
> >>ion
> >> > 1    1   -1    0    1    1    1    1
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.master.testdistributedlogsplitting.testdelayeddel
> >>ete
> >> > onfailure    1    1   -1    0    1    1    1    1
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.master.testmasterfailover.testmasterfailoverwithm
> >>ock
> >> > edritondeadrs    1    1    1    1    1   -1    0    1
> >> > org.apache.hadoop.hbase.master.testtablelockmanager.testdelete    1
> >> 1
> >> > -1    0    1    1    1    1
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.procedure.testzkprocedurecontrollers.testsimplezk
> >>coh
> >> > ortmembercontroller    1    1    1    1    1    1   -1    0
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testex
> >>ist
> >> > ingznodeblockssplitandwerollback    1    1    1    1    1   -1    0
> >> 1
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testsh
> >>oul
> >> >
> >>
> >>dthrowioexceptionifstorefilesizeisemptyandshouldsuccessfullyexecuterollba
> >>ck
> >> > 1    1   -1    0    1    1    1    1
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testsh
> >>utd
> >> > ownfixupwhendaughterhassplit   -1   -1   -1    0   -1    0   -1    0
> >> > org.apache.hadoop.hbase.regionserver.wal.testhlog.testlogcleaning    0
> >> -1
> >> > -1    0    1    1    1    1
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.replication.testmultislavereplication.testmultisl
> >>ave
> >> > replication    1    1   -1    0    1    1    1    1
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.replication.testreplicationqueuefailover.queuefai
> >>lov
> >> > er    1    1    1   -1    0   -1    0    1
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.replication.testreplicationqueuefailovercompresse
> >>d.q
> >> > ueuefailover    1    1    1   -1    0   -1    0    1
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.security.access.testaccesscontrolfilter.testquali
> >>fie
> >> > raccess   -1    0    1   -1    0    1    1    1
> >> > org.apache.hadoop.hbase.util.testhbasefsck.testdegenerateregions    1
> >> -1
> >> > 0    1    1    1    1    1
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.util.testhbasefsck.testfixassignmentsandnohdfsche
> >>cki
> >> > ng    1   -1    0    1    1    1    1    1
> >> > org.apache.hadoop.hbase.util.testhbasefsck.testfixbytable    1   -1
> >> 0
> >> > 1    1    1    1    1
> >> > org.apache.hadoop.hbase.util.testhbasefsck.testhbckthreadpooling    1
> >> -1
> >> > 0    1    1    1    1    1
> >> > org.apache.hadoop.hbase.util.testhbasefsck.testlingeringreferencefile
> >>  1
> >> > -1    0    1    1    1    1    1
> >> > org.apache.hadoop.hbase.util.testhbasefsck.testmissingfirstregion    1
> >> -1
> >> > 0    1    1    1    1    1
> >> > org.apache.hadoop.hbase.util.testhbasefsck.testmissinglastregion    1
> >> -1
> >> > 0    1    1    1    1    1
> >> > org.apache.hadoop.hbase.util.testhbasefsck.testnotinhdfs    1   -1
> >>0
> >> >  1
> >> > 1    1    1    1
> >> > org.apache.hadoop.hbase.util.testhbasefsck.testnotinmetaordeployedhole
> >>  1
> >> > -1    0    1    1    1    1    1
> >> > org.apache.hadoop.hbase.util.testhbasefsck.testoverlapandorphan    1
> >> -1
> >> > 0    1    1    1    1    1
> >> > org.apache.hadoop.hbase.util.testhbasefsck.testregionhole    1   -1
> >> 0
> >> > 1    1    1    1    1
> >> >
> >>org.apache.hadoop.hbase.util.testhbasefsck.testregionshouldnotbedeployed
> >> > 1   -1    0    1    1    1    1    1
> >> > org.apache.hadoop.hbase.util.testhbasefsck.testsidelineoverlapregion
> >>  1
> >> > -1    0    1    1    1    1    1
> >> > org.apache.hadoop.hbase.util.testhbasefsck.testsplitdaughtersnotinmeta
> >>  1
> >> > -1    0    1    1    1    1    1
> >> >
> >> > More test cases are not in good shape.
> >> >
> >> > HBase-0.94(from last 10 runs)
> >> >
> >> > Failed Test Cases               873  874  875  876  877  878  879  880
> >>  881
> >> > ========================================================
> >> > org.apache.hadoop.hbase.backup.testhfilearchiving.testcleaningrace
> >>-1
> >> >  0
> >> > 1    1    1    1    1    1    1
> >> > org.apache.hadoop.hbase.catalog.testmetareadereditor.testgetregion
> >>1
> >> >  1
> >> > -1    0    1    1    1    1    1
> >> > org.apache.hadoop.hbase.catalog.testmetareadereditor.testretrying    1
> >>  1
> >> > -1    0    1    1    1    1    1
> >> >
> >>org.apache.hadoop.hbase.catalog.testmetareadereditor.testscanmetafortable
> >> > 1    1   -1    0    1    1    1    1    1
> >> > org.apache.hadoop.hbase.catalog.testmetareadereditor.testtableexists
> >>  1
> >> > 1   -1    0    1    1    1    1    1
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.client.testmetamigrationremovinghtd.org.apache.ha
> >>doo
> >> > p.hbase.client.testmetamigrationremovinghtd    0   -1    0    0    0
> >>  0
> >> > 0    0    0
> >> >
> >>org.apache.hadoop.hbase.client.testmultiparallel.testflushcommitsnoabort
> >> > 1    1    1    1    1   -1    0    1    1
> >> >
> >>
> >>org.apache.hadoop.hbase.client.testmultiparallel.testflushcommitswithabor
> >>t
> >> > 1    1    1    1    1   -1    0    1    1
> >> > org.apache.hadoop.hbase.client.testshell.testrunshelltests    1    1
> >>  1
> >> > 1    1    1   -1    0    1
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.master.testmasterfailover.testmasterfailoverwithm
> >>ock
> >> > edritondeadrs    1    1   -1   -1    0    1    1    1    1
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.regionserver.testatomicoperation.testmultirowmuta
> >>tio
> >> > nmultithreads    1    1    1   -1    0    1    1    1    1
> >> >
> >> >
> >>
> >>org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testsh
> >>utd
> >> > ownfixupwhendaughterhassplit    1   -1    0    1    1    1    1    1
> >>  1
> >> >
> >> > As you can see 0.94 is stable and testmetamigrationremovinghtd  hasn't
> >> been
> >> > run for a while either being removed or due to other failures. Some
> >>test
> >> > cases aren't stable but not too bad.
> >> >
> >> > Thanks,
> >> > -Jeffrey
> >> >
> >> >
> >> >
> >> >
> >>
> >> --
> >> Best regards,
> >>
> >>    - Andy
> >>
> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> >> (via Tom White)
> >>
>
>
>

--001a11c36e004c2b8404e2d74c20--