Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of paul.joseph.davis@gmail.com
 designates 209.85.220.178 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <51547C60.3000603@apache.org>
References: <51535F0B.7060408@apache.org> <51547C60.3000603@apache.org>
From: Paul Davis <paul.joseph.davis@gmail.com>
Date: Fri, 29 Mar 2013 13:19:41 -0500
Message-ID: 
 <CAJ_m3YCM=96AWx8NCagsSQ+RxTOje84LwesHxAS4vZtvj9C+Bw@mail.gmail.com>
Subject: Re: Javascript Test Suite
To: dev@couchdb.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Awesome work Wendall.

For #2 I did some work during the merge stuff to try and make the
server restarts more robust. Its a pretty small commit but definitely
seemed to help.

https://github.com/davisp/couchdb/commit/0cbf6a9cea01eea599524bcdb77dedb322c7ade4

On Thu, Mar 28, 2013 at 12:22 PM, Wendall Cada <wendallc@apache.org> wrote:
> On 03/27/2013 02:05 PM, Wendall Cada wrote:
>>
>> In 1.3.0, there is a new part of the test suite to run the javascript
>> tests from the command line. I'm running into various issues on different
>> hardware/OS configurations. Mostly, tests hanging or timing out and failing.
>> These are really hard to troubleshoot, as they all pass just fine if run
>> individually.
>>
>> What I'm experimenting with today is rewriting how the tests are
>> implemented to be run one at a time from a loop in bash, versus a loop in
>> javascript. I think the failures I'm running into are improper
>> setup/teardown. There may be an issue with rapid delete and adding a db, or
>> rapidly starting and stopping couchdb, but I think this is not what's
>> happening in my failures.
>>
>> The nature of spidermonkey doesn't allow for spawning threads, or
>> sandboxing, etc, so it's hard looking at the test suite to see how I can
>> improve running all tests. I think it's far better to have the setup spawn a
>> new interpreter for each test. Tear down will kill the interpreter.
>>
>> Wendall
>
> An update here. I spent some time with this yesterday.
>
> There are two problems here I'd like to address.
>
> 1. When running `make check`, the entire javascript test suite runs in a
> single interpreter thread. This means that if there are any broken tests,
> then entire suite bails, as it can crash the interpreter. As I said, to
> address this, I simply want to refactor the test suite to spawn a new
> interpreter thread for every test. This is working well for me with my quick
> work yesterday, and allows me to put a small delay in between tests to
> ensure proper tear down from the previous test.
>
> 2. Some of the tests are written in a way that can cause a race condition in
> the test itself. Quite a few places exist in the current javascript tests
> where infinite loops can get generated. I simply want to re-factor several
> of these tests to work as expected and avoid the race conditions entirely.
>
> One additional thought I had was to mimic the output of the etap tests a bit
> more. Show the entire path to the test file being run, with a status at the
> end of the line. Change the individual test run to include the full path to
> the test to be run. Additionally, the run scripts would benefit from having
> some help output. I'm in the process of gathering notes for the
> documentation as well.
>
> My thinking is that if the test suites are similar, it will be helpful to
> new developers in troubleshooting issues.
>
> If nobody has any objections, I'd like to get started on this work. I have a
> local branch in progress for this already, but it's mostly just a scratch
> space for testing so far.
>
> Wendall