couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Noah Slater <nsla...@apache.org>
Subject Re: [VOTE] Release Apache CouchDB 1.3.0-rc.1
Date Tue, 12 Mar 2013 19:40:15 GMT
On 12 March 2013 12:19, Wendall Cada <wendallc@83864.com> wrote:

> My issue is that the process for a typically release takes me about ~2 hrs
> to fully test, build packages and vote. I spent about 6 hours yesterday
> (that I really didn't have, but really care about CouchDB), and still was
> unable to get a successful rpm to build because test failures. These tests
> also take a very long time to complete, so re-running in a loop until they
> pass is a tough proposition.
>

This is exactly the problem I was facing as the release manage. And this is
the reason I decided to split the tests out from the release candidate
preparation stage. Instead, the tests are moved to the release testing
phase.

I am not sure what to do about this. There seems to be three compounding
issues: there are a lot of tests, the tests take a very long time, and the
tests are flakey. For a build that requires every single one of the tests
to pass, this can mean you're sitting with it for 6 hours or more until you
get a run that happens to have 100% test success. Obviously, this needs to
be fixed.

If we can verify that the tests do in fact pass, when you run
them separately  then my position is that we should release 1.3.0 and then
focus some time and attention on making sure that we're in a better
position for the next release, etc.

That is, considering these major test suite bugs, but not release critical
bugs.

I think one of the things we could do, perhaps after this realise, so as
not to delay it, is actually collect a statistical sample of test results.
i.e. Run the entire test suite like 1000 times on one of these machines.
>From that, we ought to be able to identify the "worst offenders" and then
we can focus our efforts on fixing those up.

Does that sound like a sensible plan? I can work with you on that if you
like?


> What I'd hate to see is a downstream release person getting this,
> expecting to spend an hour on it, and having it eat half their day. Also,
> the current builds break my "production ready" criteria internally, and I
> think this will be the same for others.
>
> While I fully realize that my vote won't slow this from being released,
> I've personally been unable to actually use 1.3.0, as I'm still stuck
> trying to get tests to fully run.
>

I see what you're saying, and I really want to make life easier for
our down-streams  In this instance though, I feel like the project is stuck
between a rock and a hard place. We're 6 months late releasing 1.3.0. Our
tests are causing problems.

I am currently in in favour of just shipping 1.3.0, buggy test suite and
all, and trying to fix the tests separately. I'd rather see us shipping
releases with buggy tests, than waiting another three months (or whatever)
for the tests to get better.


> As for detailed information, I started another thread for this, and I can
> give any additional details necessary as they are requested. Additionally,
> I'll spend some time today mucking about with the tests a bit to see if I
> can get something useful for an actual report.
>

Cool. What do you think of my idea about running this n times and
collecting a statistical distribution of failure?

I was reading an article recently about CI systems, and this guy was
showing off (rightly so) that their tests are SO reliable, they fail in
like 1 out of 1 billion runs or something completely absurd and annoyingly
great.

-- 
NS

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message