hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15651) Track our flaky tests and use them to improve our build environment
Date Fri, 15 Apr 2016 05:34:25 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15242470#comment-15242470
] 

stack commented on HBASE-15651:
-------------------------------

Script is missing apache license.

Does this if I pass it general Trunk URL:

kalashnikov:hbase.git stack$ python ./dev-support/flakies.py https://builds.apache.org/view/All/job/HBase-Trunk_matrix/
Getting test results for https://builds.apache.org/view/All/job/HBase-Trunk_matrix/848/
Traceback (most recent call last):
  File "./dev-support/flakies.py", line 36, in <module>
    run_id_to_results[run["number"]] = get_build_results(run["url"])
  File "./dev-support/flakies.py", line 20, in get_build_results
    for test_cases in json_response["suites"]:
KeyError: 'suites'

Otherwise, it is excellent.  I can check it in and fix the apache license if you want or do
you want to do it and address the above (tell user what URL you are expecting?)

We should probably remove the ./dev-tools/jenkins-tools/  They used to do something like this:

{code}
 22 A tool which pulls test case results from Jenkins server. It displays a union of failed
test cases
 23 from the last 15(by default and actual number of jobs can be less depending on availablity)
runs
 24 recorded in Jenkins sever and track how each of them are performed for all the last 15
runs(passed,
 25 not run or failed)
{code}

Here is an example run:

{code}
kalashnikov:hbase.git stack$ python ./dev-support/flakies.py https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/
Getting test results for https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/848/
Getting test results for https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/847/
Getting test results for https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/846/
Getting test results for https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/845/
No test results for https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/845/
Getting test results for https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/844/
Getting test results for https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/843/
Getting test results for https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/842/
Getting test results for https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/841/
Getting test results for https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/840/
Getting test results for https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/839/
Getting test results for https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/838/
Getting test results for https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/837/
Getting test results for https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/836/
Getting test results for https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/835/
Getting test results for https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/834/
Getting test results for https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/833/
Getting test results for https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/832/
Getting test results for https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/831/
Getting test results for https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/830/
Getting test results for https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/829/
Getting test results for https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=latest1.7,label=yahoo-not-h2/745/
                                                                                         
 Test Name  Failed  Total Runs  Flakyness
                                               client.TestMetaCache#testPreserveMetaCacheOnException
          1      18   6%
                                               ipc.TestAsyncIPC#testRTEDuringAsyncConnectionSetup[0]
          1      20   5%
              security.visibility.TestVisibilityLabelsWithACL#testVisibilityLabelsForUserWithNoAuths
          1      19   5%
                                 regionserver.TestRegionMergeTransactionOnCluster#testWholesomeMerge
          2      19  11%
                                            coprocessor.TestMasterObserver#testTableNamesEnumeration
          1      19   5%
                                    client.TestBlockEvictionFromClient#testReverseScanWithCompaction
          1      19   5%
                              regionserver.TestRegionMergeTransactionOnCluster#testMergeWithReplicas
          3      19  16%
                              io.hfile.TestScannerFromBucketCache#testBasicScanWithOnheapBucketCache
          1      19   5%
                      master.procedure.TestMasterFailoverWithProcedures#testWALfencingWithWALRolling
          5      19  26%
                                                  coprocessor.TestMasterObserver#testTableOperations
          1      19   5%
                                                          coprocessor.TestMasterObserver#testStarted
          1      19   5%
                                  regionserver.TestFailedAppendAndSync#testLockupAroundBadAssignSync
          1      19   5%
             security.visibility.TestVisibilityLabelsWithACL#testScanForSuperUserWithFewerLabelAuths
          1      19   5%
                security.visibility.TestVisibilityLabelsWithACL#testLabelsTableOpsWithDifferentUsers
          1      19   5%
           master.balancer.TestStochasticLoadBalancer2#testRegionReplicasOnMidClusterHighReplication
          2      19  11%
                            master.procedure.TestMasterFailoverWithProcedures#testCreateWithFailover
          5      19  26%
             regionserver.throttle.TestFlushWithThroughputController#testFlushControlForStripedStore
          2      19  11%
{code}

> Track our flaky tests and use them to improve our build environment
> -------------------------------------------------------------------
>
>                 Key: HBASE-15651
>                 URL: https://issues.apache.org/jira/browse/HBASE-15651
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Appy
>            Assignee: Appy
>         Attachments: flakies.py
>
>
> So i have written this simple script (attached) which looks at history of the [post commit
build|https://builds.apache.org/view/All/job/HBase-Trunk_matrix] and outputs a list of flaky
tests with some numbers.
> Next steps:
> 1. Setup a jenkins job (say *find-flaky-tests*) to run this script daily. We can either
directly pull these results into other jobs using curl on this job's artifacts, or commit
the list of flaky test to repo (idk if it's possible to commit something from jenkins job).
> We'll collect results from both *post-commit* job (to add new flakies) and *flaky-tests*
job (to delete tests which are no more flaky).
> 2. Change *pre-commit* and *post-commit* jobs to ignore these tests using --exclude maven
flag. Someone familiar with yetus might be able to do it easily.
> 3. Setup a new job (say *flaky-tests*) to run only these flaky tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message