couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Lehnardt <...@apache.org>
Subject Re: The JavaScript Test Suite
Date Mon, 12 Oct 2015 11:26:53 GMT

> On 12 Oct 2015, at 13:15, Dale Harvey <dale@arandomurl.com> wrote:
> 
>> - instead of fixed test db names like `test_suite_db` that are used in
> almost every test,
>> we now generate a random database name for each test. The JS tests
> expected db
>> deletion to be synchronous, and that’s no longer the case in 2.0. Instead
> of
>> error-prone polling to wait for db deletion, we just use new db for every
> test.
>> An added benefit now is that (some) tests can run in parallel.
> 
> Little confused by this, PouchDB uses consistent names in its test suite
> (we had random
> ones for a while and moved away), we very much depend on the database being
> deleted
> on the request returning and dont seem to have any problems with it against
> master.

Does the PouchDB test suite runs against the 5986 port, or with an n=1 cluster?
This is with the default n=3 cluster against 5986.

Best
Jan
--

> 
> On 12 October 2015 at 12:21, Jan Lehnardt <jan@apache.org> wrote:
> 
>> 
>>> On 11 Oct 2015, at 18:53, Sebastian Rothbucher <
>> sebastianrothbucher@googlemail.com> wrote:
>>> 
>>> Hi Jan,
>>> 
>>> I transferred some of my findings to your branch - pls. find a PR for the
>>> first few tests submitted 2 your branch. I put in TODOs where either
>> there
>>> is most probably a bug (filed COUCHDB-2852 and COUCHDB-2853) or we have
>> to
>>> do some more work (like making config available).
>> 
>> Excellent, thanks! :)
>> 
>> 
>>> Will from the Pouch team had the great idea of using -n 1 to mitigate
>>> Heisenbugs due to distribution. I adopted it and it did work quite well.
>> It
>>> might make sense to put this into the dev/run call as a general rule,
>> esp.
>>> when checking for 201 (VS 202) status codes. Probably you know more on
>>> that.
>> 
>> We should carefully evaluate this :)
>> 
>> 
>>> Anyways: I think we have to keep using _config on the local port as many
>>> tests (like auth tests) depend on changed configurations. BTW: w/ -n 1
>> it's
>>> only one call, but calling 3 _config(s) migt also work.
>>> Hopefully (I just arrived at some test w/ that recently) we can spare
>>> ourselves the _compact thing. I think we'd have to compact every single
>>> shard via local port, right? for the first tests, we can go w/out as it
>>> seems. Maybe we should stick to that for a while.
>> 
>> My concern with this is that as far as I care, the local port is irrelevant
>> to how CouchDB works. Of course it isn’t 100% irrelevant, but the cluster
>> port is our publicised API for 2.0 and it’s the one that replaces the 1.x
>> API for people upgrading. So having any tests that run on the local port
>> of 2.0 don’t help us in any way (that’s not to say it still useful to have
>> tests against them for the moment).
>> 
>> Many of the _config-dependent tests substitute out the _users or
>> _replicator
>> databases, maybe we can find a workaround for those by just using the stock
>> databases.
>> 
>> For the other tests, maybe we split them out into their own “test suites”
>> that start their own test cluster in the configuration they need and shut
>> it down afterwards.
>> 
>> Best
>> Jan
>> --
>> 
>> 
>>> 
>>> Looking fw 2 your thoughts
>>> 
>>> Best
>>>   Sebastian
>>> 
>>> P.S::  Maybe we can adopt some stuff from
>>> 
>> https://github.com/apache/couchdb/compare/master...sebastianrothbucher:clustertest
>>> 
>>> P.P.S.: Will keep working on it, just not in the next few days,
>>> unfortunately ;-(
>>> 
>>> On Sun, Oct 11, 2015 at 12:00 PM, Jan Lehnardt <jan@apache.org> wrote:
>>> 
>>>> (I need your help deciding what to do with a bunch of tests, jump to
>>>> “Discussion” below if you don’t care about the details before.)
>>>> 
>>>> * * *
>>>> 
>>>> Hi all,
>>>> 
>>>> I spent some more time getting the 1.x JavaScript tests running on the
>>>> cluster port of 2.0. WIP here:
>> https://github.com/apache/couchdb/pull/354
>>>> — I’ve gotten a lot further, but there is still a lot to do and everyone
>>>> here can help, I’ve posted instructions on how to do so further down. No
>>>> Erlang knowledge needed, this is ideal for folks who want to start with
>>>> contributing to CouchDB.
>>>> 
>>>> 
>>>> Notable changes:
>>>> 
>>>> - instead of fixed test db names like `test_suite_db` that are used in
>>>> almost every test, we now generate a random database name for each test.
>>>> The JS tests expected db deletion to be synchronous, and that’s no
>> longer
>>>> the case in 2.0. Instead of error-prone polling to wait for db
>> deletion, we
>>>> just use new db for every test. An added benefit now is that (some)
>> tests
>>>> can run in parallel.
>>>> 
>>>> - tests now clean up after themselves (incomplete): tests previously
>> only
>>>> collectively used three or four databases, and deleted and re-created
>> them
>>>> before each test was run. Now with the change of creating a new database
>>>> per test, we also need to clean up databases at the end of a test,
>> because
>>>> otherwise we a) leave a lot of databases around and b) CouchDB runs out
>> of
>>>> file descriptors during the test run. The cleanup isn’t 100% done yet,
>> so
>>>> we should check for stray databases after JS test runs until we’ve got
>> it
>>>> cleaned. One particular aspect here is when a test fails, it never
>> reaches
>>>> the cleanup code, so we might have to wrap everything into a try/catch
>>>> block.
>>>> 
>>>> - I disabled all tests that still fail and made them so they print the
>>>> reason why they are failing, so that fixing them should be a lot easier
>>>> now. Some of them print just “TODO”, their error cause is yet to be
>>>> determined.
>>>> 
>>>> - couchdb.query() which previously executed temporary views, now under
>> the
>>>> hood creates a design doc with a random name and queries that view. This
>>>> fixes all tests relying on temp views without having to change them all
>>>> individually. This is because 2.0 doesn’t have temp views anymore
>> (which is
>>>> a good thing).
>>>> 
>>>> - restartServer() is no longer used (mostly).
>>>> 
>>>> I already filed a few bugs that showed things missing in the cluster API
>>>> that we’d expect to be there:
>>>> 
>>>> - https://issues.apache.org/jira/browse/COUCHDB-2849
>>>> - https://issues.apache.org/jira/browse/COUCHDB-2850
>>>> - https://issues.apache.org/jira/browse/COUCHDB-2851
>>>> 
>>>> Based on this I am ever more convinced we should get this test suite
>> work
>>>> against the 2.0 clustered API, because we’ll be finding some more
>>>> inconsistencies that we should fix before we invite beta testers.
>>>> 
>>>> 
>>>> ## Discussion:
>>>> 
>>>> We have a whole number of tests that use the test-internal function
>>>> run_on_modified_server. They use the _config API to test CouchDB in
>>>> different configurations from the default one. All of these tests now
>> throw
>>>> an error that _config isn’t available on the clustered API. How should
>> we
>>>> handle them?
>>>> 
>>>> The tests are:
>>>> 
>>>> - erlang_views.js
>>>> - jsonp.js
>>>> - oauth_users_db.js
>>>> - oauth_users_db.js
>>>> - proxyauth.js
>>>> - reader_acl.js
>>>> - rewrite.js
>>>> - security_validation.js
>>>> - show_documents.js
>>>> - users_db.js
>>>> - view_errors.js
>>>> 
>>>> 
>>>> And a few tests aside from the proper compaction tests call _compact
>> which
>>>> is also not available on the cluster. What to do with them?
>>>> 
>>>> - recreate_doc.js
>>>> - and a few more that are currently disabled for other reasons, sorry I
>>>> don’t have a full list at this point.
>>>> 
>>>> * * *
>>>> 
>>>> 
>>>> Bottom line: It’d be great if other folks here could join in the fun. No
>>>> Erlang needed, this is purely JavaScript and CouchDB HTTP API-work :)
>>>> 
>>>> 
>>>> ## How to help:
>>>> 
>>>> 1. Check out https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip
>>>> 2. run `make clean && ./configure -c --disable-docs --disable-fauxton
&&
>>>> make`
>>>> 3. `make javascript` runs the whole test suite
>>>> 
>>>> To run the whole test suite without waiting for rebar to confirm that no
>>>> Erlang code needs to be compiled, run `./test/javascript/run` for the
>> whole
>>>> test suite, or `./test/javascript/run all_docs` to run an individual
>> test
>>>> (all_docs runs test/javascript/tests/all_docs.js, other file names match
>>>> accordingly).
>>>> 
>>>> Feel free to send me PRs against the branch on
>>>> https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip, I’ll merge
>>>> them, so the PR on apache/couchdb gets updated.
>>>> 
>>>> I’m looking forward to your contributions! :)
>>>> 
>>>> If you have any questions or have trouble getting started or just would
>>>> like some hand-holding, let me know!
>>>> 
>>>> 
>>>> Best
>>>> Jan
>>>> --
>>>> 
>>>> 
>>>> 
>>>> 
>> 
>> --
>> Professional Support for Apache CouchDB:
>> http://www.neighbourhood.ie/couchdb-support/
>> 
>> 

-- 
Professional Support for Apache CouchDB:
http://www.neighbourhood.ie/couchdb-support/


Mime
View raw message