Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of paul.joseph.davis@gmail.com
 designates 209.85.220.180 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <CB250FE6-78D7-4308-A54A-0A482940D9CB@apache.org>
References: 
 <CAJ_m3YDr_G34C5SK=61hbkNWCP52=F46odGHn7wam7eU3cBktQ@mail.gmail.com>
 <EB43EB02-71C0-40C8-9852-1B573DB3BB23@dionne-associates.com>
 <CAJ_m3YD56cSuuhD_3Vv7SVsJrXw-_jyk8CUWms22FHLOZUjBLg@mail.gmail.com>
 <CB250FE6-78D7-4308-A54A-0A482940D9CB@apache.org>
From: Paul Davis <paul.joseph.davis@gmail.com>
Date: Tue, 9 Aug 2011 10:19:11 -0500
Message-ID: 
 <CAJ_m3YANYcxmMbvwGVfkFCJqvdnzAdtJkiU8HBr=L=VSLzFezg@mail.gmail.com>
Subject: Re: Futon Test Suite
To: dev@couchdb.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Tue, Aug 9, 2011 at 9:38 AM, Adam Kocoloski <kocolosk@apache.org> wrote:
> On Aug 9, 2011, at 4:48 AM, Paul Davis wrote:
>
>> On Tue, Aug 9, 2011 at 2:40 AM, Robert Dionne
>> <dionne@dionne-associates.com> wrote:
>>>>
>>>>
>>>> Also, I've been thinking more and more about beefing up the JavaScript
>>>> test suite runner and moving more of our browser tests over to
>>>> dedicated code in those tests. If anyone's interested in hacking on
>>>> some C and JavaScript against an HTTP API, let me know.
>>>
>>>
>>> Paul,
>>>
>>> =A0Jan and I talked about this a few times and I started a branch[1] al=
ong that idea. So far all I did was make a copy of the then current Futon t=
ests into
>>> test/javascript/test =A0and started looking at the small handful that f=
ail.
>>>
>>> =A0 The browser tests are great (any test is good) but they have too ma=
ny browser =A0dependent quirks, or at least I assume that because of the pl=
easant surprise
>>> one gets when they all run. So I think the goal of these runner tests w=
ould be some sort of official HTTP API suite that's part of "make check". W=
ould you agree?
>>> If so I'm happy to take this on.
>>>
>>> =A0 Also I've found eunit to be helpful in BigCouch and wondering how h=
ard it would be to support eunit in couchdb. Having tests in the modules is=
 very good for not
>>> only testing but also to help with reading and understanding what the c=
ode does.
>>>
>>> Bob
>>>
>>>
>>> [1] https://github.com/bdionne/couchdb/tree/cli-tests
>>>
>>>
>>
>> Bob,
>>
>> Exactly what I was thinking. By having our test suite have as little
>> code between the socket and the the test as possible we can ensure
>> that the tests are testing CouchDB and not some random change in the
>> behavior of our favorite web browser. I would definitely expect these
>> tests to be part of make check and hence part of the release
>> procedure.
>>
>> My current half formed thoughts are to basically split our test suite
>> into two halves. Tests that are in Erlang and are testing internals,
>> and tests that go through the HTTP interface. I love me some Erlang,
>> but I've not been think of an elegant way to make it easy to run lots
>> of HTTP tests.
>>
>> As to eunit, I'm not sure. I'm really not a huge fan of it, especially
>> mixing implementation and test code. I know rebar can separate them,
>> so its at least possible to get around that. I'd like to have a
>> unified environment for Erlang tests though. And TAP at seems like
>> it'd be easier to interface with non-Erlang tooling if we ever get
>> around to build matrices and what not. But I'm not opposed to it on
>> religious grounds if that's what people want to contribute.
>
> OTP ships with eunit_surefire[1] so interfacing with general test infrast=
ructures isn't really an issue. =A0I had to dig a little deeper to find a d=
ecent TAP consumer for Jenkins and ended up executing the tests from a Perl=
 script using TAP::Harness::JUnit. =A0I'll grant that the TAP output is eas=
ier for a human to digest than the eunit output.
>
> I imagine testing the private module functions is a topic for a good flam=
ewar. =A0I find it useful, but maybe that's a sign I've got too much logic =
in a particular module. =A0Either way, =A0a viable alternative to writing u=
nit tests that execute in the browser is something this project really need=
s. =A0Cheers,
>
> Adam
>
> [1]: http://www.erlang.org/doc/man/eunit_surefire.html

Way to totally destroy on of my last hopes of not using eunit. Thanks for t=
hat.

Also, yes. I've finally become irritated enough with clearing the
browser cache between every test that I feel its time to do something
productive about it.