couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joan Touzet <woh...@apache.org>
Subject Re: Script for displaying test errors and timing information
Date Thu, 12 Dec 2019 21:28:40 GMT
+1. I think this is useful information to collect on every CI run and 
store alongside the run, though keeping in mind we only keep the last 
7-10 runs in Jenkins nowadays.

If we need it longer, we should stash the output with the log stasher. 
This could allow graphing if someone wants to build a neato D3 thing 
that hits the couchdb-vm2 instance for stats.

On 2019-12-12 3:51 p.m., Paul Davis wrote:
> Hey all,
> 
> I was poking around at Jenkins the other day trying to get a good idea
> of how much time we're spending in various parts of the build. It
> occurred to me that one good way to at least investigate our eunit
> test suite is to parse all of the generated surefire reports. I spent
> an hour or so yesterday throwing a python script together to see what
> that might look like.
> 
> I've posted an example output and the script itself here:
> 
> https://gist.github.com/davisp/7064c1ef0dc94a99c739729b97fef10e
> 
> So far it shows some aggregate timing information along with the top
> few results for total suite time, total setup/teardown time, and
> longest individual test times. Not included in the output, but any
> test failures or skipped tests are also included in the output which
> has the nice benefit of showing exactly which test or tests failed
> during a run.
> 
> So far a few interesting things jump out. First, we spend more time in
> setup/teardown fixtures than we do running actual tests which I found
> to be rather surprising. Not so surprising is that our property tests
> are taking the longest for each run.
> 
> I thought it might be interesting to include a version of this script
> (with perhaps a somewhat more succinct output) at the end of every
> eunit run (or perhaps failed eunit run) to list out the errors for
> easier debugging of failed CI runs. And then beyond that, having it in
> the source tree would allow devs to poke around and trying to speed up
> some of the slower tests.
> 
> Another thought was to take a look at moving the property based tests
> to a nightly build and then actually increasing their run times to
> cover more of the state space for each of those tests. That way we'd
> be doing a bit more of the normal property based testing where its
> more about long soaks to find edge cases rather than acceptance
> testing for a particular change.
> 
> So far I've just thrown this together. If there's enough of a
> consensus that we'd like to see something of this nature I can work a
> bit more on improving the script to have something that's useful both
> locally for reducing the test suite times and also for spotting errors
> that failed a CI run.
> 
> Paul
> 

Mime
View raw message