couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Dionne <dio...@dionne-associates.com>
Subject Re: Test suite errors
Date Fri, 19 Mar 2010 12:05:35 GMT




On Mar 19, 2010, at 7:48 AM, Noah Slater wrote:

> I have absolutely no problem with the time taken for the tests.

cool, I misunderstood part of your issue

> 
> My only issue is that they intermittently fail. Because of that, I am now suspicious
of any results I get.
> 
> Is it really an error, or is it a timing issue or a race condition?
> 
> Suspicious tests are next to useless.

not really. In any event, looking at the code a bit couch_db:is_idle defines idle in part
as "there are now only two processes referring to this db", so it looks like it would err
on the side of concluding a db was not idle when in fact it was.

Regardless, I would not release code with a failing test suite.

> 
> On 19 Mar 2010, at 11:46, Robert Dionne wrote:
> 
>> I see similar issues, though never with 100-ref-counter.  It looks like a race condition
but should be checked because the place where it's used, couch_db:is_idle, depends on that
value being right.
>> 
>> make check is much faster that make cover	
>> 
>> I think it's ok for tests to take a long time to run and I suspect most users are
used to it. It's a measure of how solid the code is. Perhaps there could be two levels of
testing, one that's quick and superficial and sufficient to verify the build so you can run
it repeatedly in reasonable time, and the other for users at install time that includes long
running performance tests, test that run a server and so forth. At build time you'd only need
to run this once at the end.
>> 
>> 
>> 
>> On Mar 19, 2010, at 7:13 AM, Noah Slater wrote:
>> 
>>> Some of the test suites rely on timing delays, and these are unpredictable, resulting
in non-deterministic test failures. The full tag/build/test cycle is long enough as it is
- but having to start again from scratch when the last, and second, run of the test suite
fails adds a significant amount of friction for me. I would like to ask that this issue is
address as soon as possible. It is entirely my fault that this release has been delayed as
much as it has, but my job would be made significantly easier if the test suite behaved consistently.
>>> 
>>> I got the error included below this morning, and when I ran it again, there was
no error. I am going to ignore this for now, and just call a vote on the release. But doing
so is risky. I have no idea why this failed once, and as release manager, it is my duty to
understand the bugs we're shipping with. I don't like being in a position where I am ignoring
them for convenience. They exist as warning beacons, primarily for me, and when I start having
to ignore them, they have utterly failed to do their job properly.
>>> 
>>> Apologies if this email sounds frustrated. I am frustrated.
>>> 
>>> I'm not finger pointing, just trying to illustrate the reasons for my belief
that this problem should be addressed as soon as possible.
>>> 
>>> ./test/etap/run
>>> /tmp/couchdb/0.11.0/test/etap/001-load........................ok            

>>> /tmp/couchdb/0.11.0/test/etap/002-icu-driver..................ok            

>>> /tmp/couchdb/0.11.0/test/etap/010-file-basics.................ok            

>>> /tmp/couchdb/0.11.0/test/etap/011-file-headers................ok            

>>> /tmp/couchdb/0.11.0/test/etap/020-btree-basics................ok            

>>> /tmp/couchdb/0.11.0/test/etap/021-btree-reductions............ok            

>>> /tmp/couchdb/0.11.0/test/etap/030-doc-from-json...............ok            

>>> /tmp/couchdb/0.11.0/test/etap/031-doc-to-json.................ok            

>>> /tmp/couchdb/0.11.0/test/etap/040-util........................ok            

>>> /tmp/couchdb/0.11.0/test/etap/041-uuid-gen....................ok            

>>> /tmp/couchdb/0.11.0/test/etap/050-stream......................ok            

>>> /tmp/couchdb/0.11.0/test/etap/060-kt-merging..................ok            

>>> /tmp/couchdb/0.11.0/test/etap/061-kt-missing-leaves...........ok            

>>> /tmp/couchdb/0.11.0/test/etap/062-kt-remove-leaves............ok            

>>> /tmp/couchdb/0.11.0/test/etap/063-kt-get-leaves...............ok            

>>> /tmp/couchdb/0.11.0/test/etap/064-kt-counting.................ok            

>>> /tmp/couchdb/0.11.0/test/etap/065-kt-stemming.................ok            

>>> /tmp/couchdb/0.11.0/test/etap/070-couch-db....................ok            

>>> /tmp/couchdb/0.11.0/test/etap/080-config-get-set..............ok            

>>> /tmp/couchdb/0.11.0/test/etap/081-config-override.............ok            

>>> /tmp/couchdb/0.11.0/test/etap/082-config-register.............ok            

>>> /tmp/couchdb/0.11.0/test/etap/083-config-no-files.............ok            

>>> /tmp/couchdb/0.11.0/test/etap/090-task-status.................ok            

>>> /tmp/couchdb/0.11.0/test/etap/100-ref-counter.................FAILED test 8 

>>> 	Failed 1/8 tests, 87.50% okay
>>> /tmp/couchdb/0.11.0/test/etap/110-replication-httpc...........ok            

>>> /tmp/couchdb/0.11.0/test/etap/111-replication-changes-feed....ok            

>>> /tmp/couchdb/0.11.0/test/etap/112-replication-missing-revs....ok            

>>> /tmp/couchdb/0.11.0/test/etap/120-stats-collect...............ok            

>>> /tmp/couchdb/0.11.0/test/etap/121-stats-aggregates............ok            

>>> /tmp/couchdb/0.11.0/test/etap/130-attachments-md5.............ok            

>>> /tmp/couchdb/0.11.0/test/etap/140-attachment-comp.............ok            

>>> /tmp/couchdb/0.11.0/test/etap/150-invalid-view-seq............ok            

>>> /tmp/couchdb/0.11.0/test/etap/160-vhosts......................ok            

>>> Failed Test                               Stat Wstat Total Fail  List of Failed
>>> -------------------------------------------------------------------------------
>>> /tmp/couchdb/0.11.0/test/etap/100-ref-cou                8    1  8
>>> Failed 1/33 test scripts. 1/456 subtests failed.
>>> Files=33, Tests=456, 74 wallclock secs (44.01 cusr +  4.38 csys = 48.39 CPU)
>>> Failed 1/33 test programs. 1/456 subtests failed.
>>> make: *** [check] Error 255
>>> 
>> 
> 


Mime
View raw message