couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Lehnardt <...@apache.org>
Subject Re: The JavaScript Test Suite
Date Mon, 12 Oct 2015 10:21:50 GMT

> On 11 Oct 2015, at 18:53, Sebastian Rothbucher <sebastianrothbucher@googlemail.com>
wrote:
> 
> Hi Jan,
> 
> I transferred some of my findings to your branch - pls. find a PR for the
> first few tests submitted 2 your branch. I put in TODOs where either there
> is most probably a bug (filed COUCHDB-2852 and COUCHDB-2853) or we have to
> do some more work (like making config available).

Excellent, thanks! :)


> Will from the Pouch team had the great idea of using -n 1 to mitigate
> Heisenbugs due to distribution. I adopted it and it did work quite well. It
> might make sense to put this into the dev/run call as a general rule, esp.
> when checking for 201 (VS 202) status codes. Probably you know more on
> that.

We should carefully evaluate this :)


> Anyways: I think we have to keep using _config on the local port as many
> tests (like auth tests) depend on changed configurations. BTW: w/ -n 1 it's
> only one call, but calling 3 _config(s) migt also work.
> Hopefully (I just arrived at some test w/ that recently) we can spare
> ourselves the _compact thing. I think we'd have to compact every single
> shard via local port, right? for the first tests, we can go w/out as it
> seems. Maybe we should stick to that for a while.

My concern with this is that as far as I care, the local port is irrelevant
to how CouchDB works. Of course it isn’t 100% irrelevant, but the cluster
port is our publicised API for 2.0 and it’s the one that replaces the 1.x
API for people upgrading. So having any tests that run on the local port
of 2.0 don’t help us in any way (that’s not to say it still useful to have
tests against them for the moment).

Many of the _config-dependent tests substitute out the _users or _replicator
databases, maybe we can find a workaround for those by just using the stock
databases.

For the other tests, maybe we split them out into their own “test suites”
that start their own test cluster in the configuration they need and shut
it down afterwards.

Best
Jan
--


> 
> Looking fw 2 your thoughts
> 
> Best
>    Sebastian
> 
> P.S::  Maybe we can adopt some stuff from
> https://github.com/apache/couchdb/compare/master...sebastianrothbucher:clustertest
> 
> P.P.S.: Will keep working on it, just not in the next few days,
> unfortunately ;-(
> 
> On Sun, Oct 11, 2015 at 12:00 PM, Jan Lehnardt <jan@apache.org> wrote:
> 
>> (I need your help deciding what to do with a bunch of tests, jump to
>> “Discussion” below if you don’t care about the details before.)
>> 
>> * * *
>> 
>> Hi all,
>> 
>> I spent some more time getting the 1.x JavaScript tests running on the
>> cluster port of 2.0. WIP here: https://github.com/apache/couchdb/pull/354
>> — I’ve gotten a lot further, but there is still a lot to do and everyone
>> here can help, I’ve posted instructions on how to do so further down. No
>> Erlang knowledge needed, this is ideal for folks who want to start with
>> contributing to CouchDB.
>> 
>> 
>> Notable changes:
>> 
>> - instead of fixed test db names like `test_suite_db` that are used in
>> almost every test, we now generate a random database name for each test.
>> The JS tests expected db deletion to be synchronous, and that’s no longer
>> the case in 2.0. Instead of error-prone polling to wait for db deletion, we
>> just use new db for every test. An added benefit now is that (some) tests
>> can run in parallel.
>> 
>> - tests now clean up after themselves (incomplete): tests previously only
>> collectively used three or four databases, and deleted and re-created them
>> before each test was run. Now with the change of creating a new database
>> per test, we also need to clean up databases at the end of a test, because
>> otherwise we a) leave a lot of databases around and b) CouchDB runs out of
>> file descriptors during the test run. The cleanup isn’t 100% done yet, so
>> we should check for stray databases after JS test runs until we’ve got it
>> cleaned. One particular aspect here is when a test fails, it never reaches
>> the cleanup code, so we might have to wrap everything into a try/catch
>> block.
>> 
>> - I disabled all tests that still fail and made them so they print the
>> reason why they are failing, so that fixing them should be a lot easier
>> now. Some of them print just “TODO”, their error cause is yet to be
>> determined.
>> 
>> - couchdb.query() which previously executed temporary views, now under the
>> hood creates a design doc with a random name and queries that view. This
>> fixes all tests relying on temp views without having to change them all
>> individually. This is because 2.0 doesn’t have temp views anymore (which is
>> a good thing).
>> 
>> - restartServer() is no longer used (mostly).
>> 
>> I already filed a few bugs that showed things missing in the cluster API
>> that we’d expect to be there:
>> 
>> - https://issues.apache.org/jira/browse/COUCHDB-2849
>> - https://issues.apache.org/jira/browse/COUCHDB-2850
>> - https://issues.apache.org/jira/browse/COUCHDB-2851
>> 
>> Based on this I am ever more convinced we should get this test suite work
>> against the 2.0 clustered API, because we’ll be finding some more
>> inconsistencies that we should fix before we invite beta testers.
>> 
>> 
>> ## Discussion:
>> 
>> We have a whole number of tests that use the test-internal function
>> run_on_modified_server. They use the _config API to test CouchDB in
>> different configurations from the default one. All of these tests now throw
>> an error that _config isn’t available on the clustered API. How should we
>> handle them?
>> 
>> The tests are:
>> 
>> - erlang_views.js
>> - jsonp.js
>> - oauth_users_db.js
>> - oauth_users_db.js
>> - proxyauth.js
>> - reader_acl.js
>> - rewrite.js
>> - security_validation.js
>> - show_documents.js
>> - users_db.js
>> - view_errors.js
>> 
>> 
>> And a few tests aside from the proper compaction tests call _compact which
>> is also not available on the cluster. What to do with them?
>> 
>> - recreate_doc.js
>> - and a few more that are currently disabled for other reasons, sorry I
>> don’t have a full list at this point.
>> 
>> * * *
>> 
>> 
>> Bottom line: It’d be great if other folks here could join in the fun. No
>> Erlang needed, this is purely JavaScript and CouchDB HTTP API-work :)
>> 
>> 
>> ## How to help:
>> 
>> 1. Check out https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip
>> 2. run `make clean && ./configure -c --disable-docs --disable-fauxton &&
>> make`
>> 3. `make javascript` runs the whole test suite
>> 
>> To run the whole test suite without waiting for rebar to confirm that no
>> Erlang code needs to be compiled, run `./test/javascript/run` for the whole
>> test suite, or `./test/javascript/run all_docs` to run an individual test
>> (all_docs runs test/javascript/tests/all_docs.js, other file names match
>> accordingly).
>> 
>> Feel free to send me PRs against the branch on
>> https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip, I’ll merge
>> them, so the PR on apache/couchdb gets updated.
>> 
>> I’m looking forward to your contributions! :)
>> 
>> If you have any questions or have trouble getting started or just would
>> like some hand-holding, let me know!
>> 
>> 
>> Best
>> Jan
>> --
>> 
>> 
>> 
>> 

-- 
Professional Support for Apache CouchDB:
http://www.neighbourhood.ie/couchdb-support/


Mime
View raw message