Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 9.0 \(3094\))
Subject: Re: The JavaScript Test Suite
From: Jan Lehnardt <jan@apache.org>
In-Reply-To: 
 <CA+pq5fqP6T_2M+rkAgVRV69NzVmor8DmozJTtrvkfMX4kd=5pw@mail.gmail.com>
Date: Mon, 12 Oct 2015 12:21:50 +0200
Content-Transfer-Encoding: quoted-printable
Message-Id: <95DEDD8F-34F2-4D3C-A037-CD787D765B00@apache.org>
References: <6443EF40-E58C-4804-869D-3BDA1132C6E9@apache.org>
 <CA+pq5fqP6T_2M+rkAgVRV69NzVmor8DmozJTtrvkfMX4kd=5pw@mail.gmail.com>
To: dev@couchdb.apache.org


> On 11 Oct 2015, at 18:53, Sebastian Rothbucher =
<sebastianrothbucher@googlemail.com> wrote:
>=20
> Hi Jan,
>=20
> I transferred some of my findings to your branch - pls. find a PR for =
the
> first few tests submitted 2 your branch. I put in TODOs where either =
there
> is most probably a bug (filed COUCHDB-2852 and COUCHDB-2853) or we =
have to
> do some more work (like making config available).

Excellent, thanks! :)


> Will from the Pouch team had the great idea of using -n 1 to mitigate
> Heisenbugs due to distribution. I adopted it and it did work quite =
well. It
> might make sense to put this into the dev/run call as a general rule, =
esp.
> when checking for 201 (VS 202) status codes. Probably you know more on
> that.

We should carefully evaluate this :)


> Anyways: I think we have to keep using _config on the local port as =
many
> tests (like auth tests) depend on changed configurations. BTW: w/ -n 1 =
it's
> only one call, but calling 3 _config(s) migt also work.
> Hopefully (I just arrived at some test w/ that recently) we can spare
> ourselves the _compact thing. I think we'd have to compact every =
single
> shard via local port, right? for the first tests, we can go w/out as =
it
> seems. Maybe we should stick to that for a while.

My concern with this is that as far as I care, the local port is =
irrelevant
to how CouchDB works. Of course it isn=E2=80=99t 100% irrelevant, but =
the cluster
port is our publicised API for 2.0 and it=E2=80=99s the one that =
replaces the 1.x
API for people upgrading. So having any tests that run on the local port
of 2.0 don=E2=80=99t help us in any way (that=E2=80=99s not to say it =
still useful to have
tests against them for the moment).

Many of the _config-dependent tests substitute out the _users or =
_replicator
databases, maybe we can find a workaround for those by just using the =
stock
databases.

For the other tests, maybe we split them out into their own =E2=80=9Ctest =
suites=E2=80=9D
that start their own test cluster in the configuration they need and =
shut
it down afterwards.

Best
Jan
--


>=20
> Looking fw 2 your thoughts
>=20
> Best
>    Sebastian
>=20
> P.S::  Maybe we can adopt some stuff from
> =
https://github.com/apache/couchdb/compare/master...sebastianrothbucher:clu=
stertest
>=20
> P.P.S.: Will keep working on it, just not in the next few days,
> unfortunately ;-(
>=20
> On Sun, Oct 11, 2015 at 12:00 PM, Jan Lehnardt <jan@apache.org> wrote:
>=20
>> (I need your help deciding what to do with a bunch of tests, jump to
>> =E2=80=9CDiscussion=E2=80=9D below if you don=E2=80=99t care about =
the details before.)
>>=20
>> * * *
>>=20
>> Hi all,
>>=20
>> I spent some more time getting the 1.x JavaScript tests running on =
the
>> cluster port of 2.0. WIP here: =
https://github.com/apache/couchdb/pull/354
>> =E2=80=94 I=E2=80=99ve gotten a lot further, but there is still a lot =
to do and everyone
>> here can help, I=E2=80=99ve posted instructions on how to do so =
further down. No
>> Erlang knowledge needed, this is ideal for folks who want to start =
with
>> contributing to CouchDB.
>>=20
>>=20
>> Notable changes:
>>=20
>> - instead of fixed test db names like `test_suite_db` that are used =
in
>> almost every test, we now generate a random database name for each =
test.
>> The JS tests expected db deletion to be synchronous, and that=E2=80=99s=
 no longer
>> the case in 2.0. Instead of error-prone polling to wait for db =
deletion, we
>> just use new db for every test. An added benefit now is that (some) =
tests
>> can run in parallel.
>>=20
>> - tests now clean up after themselves (incomplete): tests previously =
only
>> collectively used three or four databases, and deleted and re-created =
them
>> before each test was run. Now with the change of creating a new =
database
>> per test, we also need to clean up databases at the end of a test, =
because
>> otherwise we a) leave a lot of databases around and b) CouchDB runs =
out of
>> file descriptors during the test run. The cleanup isn=E2=80=99t 100% =
done yet, so
>> we should check for stray databases after JS test runs until we=E2=80=99=
ve got it
>> cleaned. One particular aspect here is when a test fails, it never =
reaches
>> the cleanup code, so we might have to wrap everything into a =
try/catch
>> block.
>>=20
>> - I disabled all tests that still fail and made them so they print =
the
>> reason why they are failing, so that fixing them should be a lot =
easier
>> now. Some of them print just =E2=80=9CTODO=E2=80=9D, their error =
cause is yet to be
>> determined.
>>=20
>> - couchdb.query() which previously executed temporary views, now =
under the
>> hood creates a design doc with a random name and queries that view. =
This
>> fixes all tests relying on temp views without having to change them =
all
>> individually. This is because 2.0 doesn=E2=80=99t have temp views =
anymore (which is
>> a good thing).
>>=20
>> - restartServer() is no longer used (mostly).
>>=20
>> I already filed a few bugs that showed things missing in the cluster =
API
>> that we=E2=80=99d expect to be there:
>>=20
>> - https://issues.apache.org/jira/browse/COUCHDB-2849
>> - https://issues.apache.org/jira/browse/COUCHDB-2850
>> - https://issues.apache.org/jira/browse/COUCHDB-2851
>>=20
>> Based on this I am ever more convinced we should get this test suite =
work
>> against the 2.0 clustered API, because we=E2=80=99ll be finding some =
more
>> inconsistencies that we should fix before we invite beta testers.
>>=20
>>=20
>> ## Discussion:
>>=20
>> We have a whole number of tests that use the test-internal function
>> run_on_modified_server. They use the _config API to test CouchDB in
>> different configurations from the default one. All of these tests now =
throw
>> an error that _config isn=E2=80=99t available on the clustered API. =
How should we
>> handle them?
>>=20
>> The tests are:
>>=20
>> - erlang_views.js
>> - jsonp.js
>> - oauth_users_db.js
>> - oauth_users_db.js
>> - proxyauth.js
>> - reader_acl.js
>> - rewrite.js
>> - security_validation.js
>> - show_documents.js
>> - users_db.js
>> - view_errors.js
>>=20
>>=20
>> And a few tests aside from the proper compaction tests call _compact =
which
>> is also not available on the cluster. What to do with them?
>>=20
>> - recreate_doc.js
>> - and a few more that are currently disabled for other reasons, sorry =
I
>> don=E2=80=99t have a full list at this point.
>>=20
>> * * *
>>=20
>>=20
>> Bottom line: It=E2=80=99d be great if other folks here could join in =
the fun. No
>> Erlang needed, this is purely JavaScript and CouchDB HTTP API-work :)
>>=20
>>=20
>> ## How to help:
>>=20
>> 1. Check out https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip
>> 2. run `make clean && ./configure -c --disable-docs --disable-fauxton =
&&
>> make`
>> 3. `make javascript` runs the whole test suite
>>=20
>> To run the whole test suite without waiting for rebar to confirm that =
no
>> Erlang code needs to be compiled, run `./test/javascript/run` for the =
whole
>> test suite, or `./test/javascript/run all_docs` to run an individual =
test
>> (all_docs runs test/javascript/tests/all_docs.js, other file names =
match
>> accordingly).
>>=20
>> Feel free to send me PRs against the branch on
>> https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip, I=E2=80=99ll =
merge
>> them, so the PR on apache/couchdb gets updated.
>>=20
>> I=E2=80=99m looking forward to your contributions! :)
>>=20
>> If you have any questions or have trouble getting started or just =
would
>> like some hand-holding, let me know!
>>=20
>>=20
>> Best
>> Jan
>> --
>>=20
>>=20
>>=20
>>=20

--=20
Professional Support for Apache CouchDB:
http://www.neighbourhood.ie/couchdb-support/