Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of paul.joseph.davis@gmail.com
 designates 209.85.219.54 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <515F64E9.9050207@apache.org>
References: <51535F0B.7060408@apache.org> <515F64E9.9050207@apache.org>
From: Paul Davis <paul.joseph.davis@gmail.com>
Date: Mon, 8 Apr 2013 09:26:00 -0500
Message-ID: 
 <CAJ_m3YAE2Ae9r8abHmFYty1-EfingB0kF3yjrAPOLv8fH9COKw@mail.gmail.com>
Subject: Re: Javascript Test Suite
To: dev@couchdb.apache.org
Content-Type: text/plain; charset=ISO-8859-1

The first bit I'd like to say is that the use of couchjs was just a
stop gap measure to get the test suite out of the browser. We used to
have to deal with so many browser issues it was just a terrible mess.
The issue with couchjs is much as you've seen that its not a very full
environment for writing tests. So just to be clear that the only real
thing tying us to that as a test platform is that we have a large
amount of JS written already so either we need to make the couchjs
better, use node, or translate tests to something that has a more
useful environment.

I've been noodling over whether we might be better off to just start
translating everything to Python or something. I've seen suggestions
for Erlang but I personally think Erlang is a terrible language for
writing tests like this (specifically, the code to test ratio is
ungood). If we had something like Python to hack on then I was also
thinking of writing a library function that would start CouchDB as a
slave process which then would remove the need to have the _restart
handler because you could just kill -9 the subprocess and restart it
with maybe a wait for when things boot again.

I reviewed your feature branch the other day and I'm +1 for pushing
that to master.

Awesome work, Wendall.

On Fri, Apr 5, 2013 at 6:57 PM, Wendall Cada <wendallc@apache.org> wrote:
> I wanted to follow up on this.
>
> I've created a feature branch for this and a JIRA issue
> https://issues.apache.org/jira/browse/COUCHDB-1762
>
> Overall, I think the worst problem is that the tests really aren't
> debuggable in any sane way, and logging is essentially useless for most
> things. The only sure way to spot an error most of the time is if it's an
> actual CouchDB bug and shows up in the log. I'm not sure how this can ever
> be fixed with the current test suite. I'd opt for testing with jasmine, but
> that would require not using couchjs for the test runner, so for now, I just
> focused on getting random failures under control.
>
> Paul was kind enough to share some code that he wrote recently to deal with
> the rampant _restart issues.
> https://github.com/davisp/couchdb/commit/0cbf6a9cea01eea599524bcdb77dedb322c7ade4
> This is a very sound approach in using a token so you can see if it actually
> restarts. The current test suite can result in false positives very easily,
> which leads to test failures. I think this is probably the biggest reason
> for the random failures. In a previous IRC conversation with Bob (rnewson),
> Jan and I think Benoit (sorry if not the case) _restart was deemed something
> that should go away. I filed a ticket for it's removal
> https://issues.apache.org/jira/browse/COUCHDB-1714, and as Bob points out in
> the comments, this is useful for the test suite. I'd argue it's only useful
> with Paul's patch adding a token. Otherwise, it's just not reliable at all.
>
> For the branch I created, instead of using _restart, I did some bash magic
> with a pipe and stop/start the process through the local run script. This
> has the same drawback of not knowing if CouchDB restarted, or we just got a
> false positive. To account for this, I put a small delay in the execution of
> the lookup, using a new method isRunning to give a little time to stop.
>
> I also changed the suite to run a new couchjs for each test file. I'm not
> certain at this point that this is even necessary at all, but I still think
> it's safer in case of a crash, since the rest of the suite can continue.
>
> Other changes I made were just timing related in running the test suite for
> spinning disks, and a couple bug fixes in individual tests.
>
> The lack of timers makes writing these tests very ugly. I really dislike
> this, but so long as the test suite needs couchjs, I don't see a way to
> avoid this without implementing our own setInterval method in C.
>
> One last item. I was getting a consistent failure in Centos 6. I tracked
> this down to a bug in libcurl. For some reason, after any xhr request that
> returns a 416, the very next send() will hang for a long time, then
> eventually crash couchjs. The specific version causing the issue is
> curl-7.19.7-35.el6 and libcurl-7.19.7-35.el6. I'm not certain if this is
> worth reporting in JIRA, but it will certainly cause a test suite failure
> consistently in attachment_ranges, but otherwise appears to be fairly
> harmless. Maybe this should be documented somewhere?
>
> Wendall
>
>
> On 03/27/2013 02:05 PM, Wendall Cada wrote:
>>
>> In 1.3.0, there is a new part of the test suite to run the javascript
>> tests from the command line. I'm running into various issues on different
>> hardware/OS configurations. Mostly, tests hanging or timing out and failing.
>> These are really hard to troubleshoot, as they all pass just fine if run
>> individually.
>>
>> What I'm experimenting with today is rewriting how the tests are
>> implemented to be run one at a time from a loop in bash, versus a loop in
>> javascript. I think the failures I'm running into are improper
>> setup/teardown. There may be an issue with rapid delete and adding a db, or
>> rapidly starting and stopping couchdb, but I think this is not what's
>> happening in my failures.
>>
>> The nature of spidermonkey doesn't allow for spawning threads, or
>> sandboxing, etc, so it's hard looking at the test suite to see how I can
>> improve running all tests. I think it's far better to have the setup spawn a
>> new interpreter for each test. Tear down will kill the interpreter.
>>
>> Wendall
>
>