hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dima Spivak (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15651) Track our flaky tests and use them to improve our build environment
Date Wed, 20 Apr 2016 02:44:25 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15249153#comment-15249153
] 

Dima Spivak commented on HBASE-15651:
-------------------------------------

Looks so awesome, Appy! For anyone who wants to see what this looks like:
{noformat}
dspivak-MBP:~ dspivak$ ./appytest.py --max_runs=15 https://builds.apache.org/job/HBase-TRUNK/
WARNING:root:Analyzing job: https://builds.apache.org/job/HBase-TRUNK/
WARNING:root:                                                                            
              Test Name  Failed  Total Runs  Flakyness
WARNING:root:      regionserver.TestRegionReplicaFailover#testSecondaryRegionKillWhilePrimaryIsAcceptingWrites[0]
          1      13   8%
WARNING:root:                                                             util.TestRegionMover#testLoadWithoutAck
          1       9  11%
WARNING:root:              regionserver.TestSplitTransactionOnCluster#testSSHCleanupDaugtherRegionsOfAbortedSplit
          1      13   8%
WARNING:root:                                     regionserver.TestRegionReplicaFailover#testPrimaryRegionKill[0]
          1      13   8%
{noformat}
Just a few more minor tweaks, but then I'm +1:
- I ran this with {{-v}} and got a lot of {{INFO:requests.packages.urllib3.connectionpool:Starting
new HTTPS connection (1): builds.apache.org}} lines because using {{logging.basicConfig}}
will trip verbosity for every module that uses logging (i.e. and not just yours). A better
way is probably to use {{logging.getLogger()}} and then work with that {{Logger}} instance.
- The output at the end can probably just use {{print}} since the intended output is stdout.
That'd also save some precious characters for those of us working on laptops. :)
- This same output should probably suppress the {{Test Name  Failed  Total Runs  Flakyness}}
header if no flaky tests were detected.
- I think there's a typo in the padding of the results (with the {{:10}} and {{:6}} reversed).
- Please change {{\-\-max_build}} to {{--max-build}} (argparse automatically makes this {{args.max_build}},
but it pretties up the command line.
- May want to output a list of which builds of a particular job were actually inspected at
the end (as it stands, we'd have to guess which ones are picked up by the script when run
in non-verbose mode).
- As a potential improvement to the user interface, we might want to entertain allowing specification
of {{--builds}} where a user would give a list of builds to inspect (instead of just going
off of the last {{n}} for a project). This doesn't block my +1 from this one, though, if it's
too much work.

> Track our flaky tests and use them to improve our build environment
> -------------------------------------------------------------------
>
>                 Key: HBASE-15651
>                 URL: https://issues.apache.org/jira/browse/HBASE-15651
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Appy
>            Assignee: Appy
>         Attachments: HBASE-15651-master-v2.patch, HBASE-15651-master.patch, flakies.py
>
>
> So i have written this simple script (attached) which looks at history of the [post commit
build|https://builds.apache.org/view/All/job/HBase-Trunk_matrix] and outputs a list of flaky
tests with some numbers.
> Next steps:
> 1. Setup a jenkins job (say *find-flaky-tests*) to run this script daily. We can either
directly pull these results into other jobs using curl on this job's artifacts, or commit
the list of flaky test to repo (idk if it's possible to commit something from jenkins job).
> We'll collect results from both *post-commit* job (to add new flakies) and *flaky-tests*
job (to delete tests which are no more flaky).
> 2. Change *pre-commit* and *post-commit* jobs to ignore these tests using --exclude maven
flag. Someone familiar with yetus might be able to do it easily.
> 3. Setup a new job (say *flaky-tests*) to run only these flaky tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message