couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Davis <paul.joseph.da...@gmail.com>
Subject Re: Script for displaying test errors and timing information
Date Fri, 13 Dec 2019 15:55:22 GMT
Cool beans. I'll take a crack at removing some of the ick around
displaying along with adding some command line arguments for
controlling output.

For long term storage I think it'd be better to store the raw XML
files rather than the formatted output which should be an easy
addition.

On Thu, Dec 12, 2019 at 3:28 PM Joan Touzet <wohali@apache.org> wrote:
>
> +1. I think this is useful information to collect on every CI run and
> store alongside the run, though keeping in mind we only keep the last
> 7-10 runs in Jenkins nowadays.
>
> If we need it longer, we should stash the output with the log stasher.
> This could allow graphing if someone wants to build a neato D3 thing
> that hits the couchdb-vm2 instance for stats.
>
> On 2019-12-12 3:51 p.m., Paul Davis wrote:
> > Hey all,
> >
> > I was poking around at Jenkins the other day trying to get a good idea
> > of how much time we're spending in various parts of the build. It
> > occurred to me that one good way to at least investigate our eunit
> > test suite is to parse all of the generated surefire reports. I spent
> > an hour or so yesterday throwing a python script together to see what
> > that might look like.
> >
> > I've posted an example output and the script itself here:
> >
> > https://gist.github.com/davisp/7064c1ef0dc94a99c739729b97fef10e
> >
> > So far it shows some aggregate timing information along with the top
> > few results for total suite time, total setup/teardown time, and
> > longest individual test times. Not included in the output, but any
> > test failures or skipped tests are also included in the output which
> > has the nice benefit of showing exactly which test or tests failed
> > during a run.
> >
> > So far a few interesting things jump out. First, we spend more time in
> > setup/teardown fixtures than we do running actual tests which I found
> > to be rather surprising. Not so surprising is that our property tests
> > are taking the longest for each run.
> >
> > I thought it might be interesting to include a version of this script
> > (with perhaps a somewhat more succinct output) at the end of every
> > eunit run (or perhaps failed eunit run) to list out the errors for
> > easier debugging of failed CI runs. And then beyond that, having it in
> > the source tree would allow devs to poke around and trying to speed up
> > some of the slower tests.
> >
> > Another thought was to take a look at moving the property based tests
> > to a nightly build and then actually increasing their run times to
> > cover more of the state space for each of those tests. That way we'd
> > be doing a bit more of the normal property based testing where its
> > more about long soaks to find edge cases rather than acceptance
> > testing for a particular change.
> >
> > So far I've just thrown this together. If there's enough of a
> > consensus that we'd like to see something of this nature I can work a
> > bit more on improving the script to have something that's useful both
> > locally for reducing the test suite times and also for spotting errors
> > that failed a CI run.
> >
> > Paul
> >

Mime
View raw message