Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of adam.kocoloski@gmail.com
 designates 209.85.218.52 as permitted sender)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: Futon Test Suite
From: Adam Kocoloski <kocolosk@apache.org>
In-Reply-To: 
 <CAJ_m3YD56cSuuhD_3Vv7SVsJrXw-_jyk8CUWms22FHLOZUjBLg@mail.gmail.com>
Date: Tue, 9 Aug 2011 10:38:09 -0400
Content-Transfer-Encoding: quoted-printable
Message-Id: <CB250FE6-78D7-4308-A54A-0A482940D9CB@apache.org>
References: 
 <CAJ_m3YDr_G34C5SK=61hbkNWCP52=F46odGHn7wam7eU3cBktQ@mail.gmail.com>
 <EB43EB02-71C0-40C8-9852-1B573DB3BB23@dionne-associates.com>
 <CAJ_m3YD56cSuuhD_3Vv7SVsJrXw-_jyk8CUWms22FHLOZUjBLg@mail.gmail.com>
To: dev@couchdb.apache.org

On Aug 9, 2011, at 4:48 AM, Paul Davis wrote:

> On Tue, Aug 9, 2011 at 2:40 AM, Robert Dionne
> <dionne@dionne-associates.com> wrote:
>>>=20
>>>=20
>>> Also, I've been thinking more and more about beefing up the =
JavaScript
>>> test suite runner and moving more of our browser tests over to
>>> dedicated code in those tests. If anyone's interested in hacking on
>>> some C and JavaScript against an HTTP API, let me know.
>>=20
>>=20
>> Paul,
>>=20
>>  Jan and I talked about this a few times and I started a branch[1] =
along that idea. So far all I did was make a copy of the then current =
Futon tests into
>> test/javascript/test  and started looking at the small handful that =
fail.
>>=20
>>   The browser tests are great (any test is good) but they have too =
many browser  dependent quirks, or at least I assume that because of the =
pleasant surprise
>> one gets when they all run. So I think the goal of these runner tests =
would be some sort of official HTTP API suite that's part of "make =
check". Would you agree?
>> If so I'm happy to take this on.
>>=20
>>   Also I've found eunit to be helpful in BigCouch and wondering how =
hard it would be to support eunit in couchdb. Having tests in the =
modules is very good for not
>> only testing but also to help with reading and understanding what the =
code does.
>>=20
>> Bob
>>=20
>>=20
>> [1] https://github.com/bdionne/couchdb/tree/cli-tests
>>=20
>>=20
>=20
> Bob,
>=20
> Exactly what I was thinking. By having our test suite have as little
> code between the socket and the the test as possible we can ensure
> that the tests are testing CouchDB and not some random change in the
> behavior of our favorite web browser. I would definitely expect these
> tests to be part of make check and hence part of the release
> procedure.
>=20
> My current half formed thoughts are to basically split our test suite
> into two halves. Tests that are in Erlang and are testing internals,
> and tests that go through the HTTP interface. I love me some Erlang,
> but I've not been think of an elegant way to make it easy to run lots
> of HTTP tests.
>=20
> As to eunit, I'm not sure. I'm really not a huge fan of it, especially
> mixing implementation and test code. I know rebar can separate them,
> so its at least possible to get around that. I'd like to have a
> unified environment for Erlang tests though. And TAP at seems like
> it'd be easier to interface with non-Erlang tooling if we ever get
> around to build matrices and what not. But I'm not opposed to it on
> religious grounds if that's what people want to contribute.

OTP ships with eunit_surefire[1] so interfacing with general test =
infrastructures isn't really an issue.  I had to dig a little deeper to =
find a decent TAP consumer for Jenkins and ended up executing the tests =
from a Perl script using TAP::Harness::JUnit.  I'll grant that the TAP =
output is easier for a human to digest than the eunit output.

I imagine testing the private module functions is a topic for a good =
flamewar.  I find it useful, but maybe that's a sign I've got too much =
logic in a particular module.  Either way,  a viable alternative to =
writing unit tests that execute in the browser is something this project =
really needs.  Cheers,

Adam

[1]: http://www.erlang.org/doc/man/eunit_surefire.html=