hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15651) Track our flaky tests and use them to improve our build environment
Date Tue, 19 Apr 2016 22:02:25 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15248766#comment-15248766
] 

stack commented on HBASE-15651:
-------------------------------

I tried it:

{code}
kalashnikov:hbase.git stack$ python ./dev-support/report-flakies.py
urls missing.

Usage: ./dev-support/report-flakies.py [options] urls

urls      Space separated list of urls (single/multi-configuration project) to analyze

Options:
-h        Prints this help message
-r        Number of runs to analyze for each job (if available in jenkins)
-mvn      Writes two strings for including/excluding these flaky tests using maven
          flags. These strings are written to files so they can be saved as artifacts
          and easily imported in other projects.
kalashnikov:hbase.git stack$
kalashnikov:hbase.git stack$ python ./dev-support/report-flakies.py  https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/
Analyzing job: https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/856/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/855/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/854/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/853/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/852/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/851/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/850/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/849/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/848/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/847/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/846/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/845/
No test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/845/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/844/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/843/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/842/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/841/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/840/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/839/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/838/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/837/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/745/
                                                                                         
 Test Name  Failed  Total Runs  Flakyness
                                        regionserver.TestDateTieredCompactionPolicy#negativeForMajor
          1      19   5%
                                               client.TestMetaCache#testPreserveMetaCacheOnException
          1      16   6%
                                  regionserver.TestFailedAppendAndSync#testLockupAroundBadAssignSync
          1      17   6%
security.visibility.TestVisibilityLabelsWithACL#testScanForUserWithFewerLabelAuthsThanLabelsInScanAuthorizations
          1      17   6%
                              regionserver.TestRegionMergeTransactionOnCluster#testMergeWithReplicas
          2      17  12%
security.visibility.TestVisibilityLabelsWithDeletes#testDeleteColumnWithLatestTimeStampWhenNoVersionMatches
          1      17   6%
                                               ipc.TestAsyncIPC#testRTEDuringAsyncConnectionSetup[0]
          2      20  10%
                              io.hfile.TestScannerFromBucketCache#testBasicScanWithOnheapBucketCache
          1      17   6%
                                                     regionserver.TestHRegion#testWritesWhileGetting
          1      17   6%
                      master.procedure.TestMasterFailoverWithProcedures#testWALfencingWithWALRolling
          3      17  18%
                                    client.TestBlockEvictionFromClient#testReverseScanWithCompaction
          1      17   6%
                                                          coprocessor.TestMasterObserver#testStarted
          1      17   6%
             regionserver.throttle.TestFlushWithThroughputController#testFlushControlForStripedStore
          1      17   6%
                                            coprocessor.TestMasterObserver#testTableNamesEnumeration
          1      17   6%
           master.balancer.TestStochasticLoadBalancer2#testRegionReplicasOnMidClusterHighReplication
          1      17   6%
                            master.procedure.TestMasterFailoverWithProcedures#testCreateWithFailover
          3      17  18%
                                                  coprocessor.TestMasterObserver#testTableOperations
          1      17   6%
Analyzing job: https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/856/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/855/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/854/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/853/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/852/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/851/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/850/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/849/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/848/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/847/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/846/
No test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/846/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/845/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/844/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/843/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/842/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/841/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/840/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/839/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/838/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/837/
Getting test results for https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.8,label=yahoo-not-h2/745/
                                                                                         
 Test Name  Failed  Total Runs  Flakyness
                                  regionserver.TestFailedAppendAndSync#testLockupAroundBadAssignSync
          1      11   9%
                                        regionserver.TestDateTieredCompactionPolicy#negativeForMajor
          2      19  11%
                                               ipc.TestAsyncIPC#testRTEDuringAsyncConnectionSetup[0]
          4      20  20%
                                    replication.regionserver.TestReplicationThrottler#testThrottling
          5      20  25%
                                                 namespace.TestNamespaceAuditor#testRegionOperations
          1      11   9%
set([u'regionserver.TestDateTieredCompactionPolicy#negativeForMajor', u'client.TestMetaCache#testPreserveMetaCacheOnException',
u'regionserver.TestFailedAppendAndSync#testLockupAroundBadAssignSync', u'security.visibility.TestVisibilityLabelsWithACL#testScanForUserWithFewerLabelAuthsThanLabelsInScanAuthorizations',
u'regionserver.TestRegionMergeTransactionOnCluster#testMergeWithReplicas', u'security.visibility.TestVisibilityLabelsWithDeletes#testDeleteColumnWithLatestTimeStampWhenNoVersionMatches',
u'ipc.TestAsyncIPC#testRTEDuringAsyncConnectionSetup[0]', u'namespace.TestNamespaceAuditor#testRegionOperations',
u'replication.regionserver.TestReplicationThrottler#testThrottling', u'io.hfile.TestScannerFromBucketCache#testBasicScanWithOnheapBucketCache',
u'regionserver.TestHRegion#testWritesWhileGetting', u'master.procedure.TestMasterFailoverWithProcedures#testWALfencingWithWALRolling',
u'client.TestBlockEvictionFromClient#testReverseScanWithCompaction', u'coprocessor.TestMasterObserver#testStarted',
u'regionserver.throttle.TestFlushWithThroughputController#testFlushControlForStripedStore',
u'coprocessor.TestMasterObserver#testTableNamesEnumeration', u'master.balancer.TestStochasticLoadBalancer2#testRegionReplicasOnMidClusterHighReplication',
u'master.procedure.TestMasterFailoverWithProcedures#testCreateWithFailover', u'coprocessor.TestMasterObserver#testTableOperations'])
{code}

Want to fix the bit on the end before commit?

> Track our flaky tests and use them to improve our build environment
> -------------------------------------------------------------------
>
>                 Key: HBASE-15651
>                 URL: https://issues.apache.org/jira/browse/HBASE-15651
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Appy
>            Assignee: Appy
>         Attachments: HBASE-15651-master.patch, flakies.py
>
>
> So i have written this simple script (attached) which looks at history of the [post commit
build|https://builds.apache.org/view/All/job/HBase-Trunk_matrix] and outputs a list of flaky
tests with some numbers.
> Next steps:
> 1. Setup a jenkins job (say *find-flaky-tests*) to run this script daily. We can either
directly pull these results into other jobs using curl on this job's artifacts, or commit
the list of flaky test to repo (idk if it's possible to commit something from jenkins job).
> We'll collect results from both *post-commit* job (to add new flakies) and *flaky-tests*
job (to delete tests which are no more flaky).
> 2. Change *pre-commit* and *post-commit* jobs to ignore these tests using --exclude maven
flag. Someone familiar with yetus might be able to do it easily.
> 3. Setup a new job (say *flaky-tests*) to run only these flaky tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message