couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Lehnardt <...@apache.org>
Subject Re: The JavaScript Test Suite
Date Wed, 14 Oct 2015 10:23:47 GMT
I’ve run the 1.x test suite against an -n1 cluster and it doesn’t improve things
significantly, although I haven’t looked at database deletes, as I’ve got these
covered for now.

So there is still some work that needs doing :)

Best
Jan
-- 

> On 12 Oct 2015, at 13:30, Dale Harvey <dale@arandomurl.com> wrote:
> 
> Started with
> 
> python dev/run -n 1 --with-admin-party-please &
> 
> and export COUCH_HOST='http://127.0.0.1:15984'
> 
> 
> On 12 October 2015 at 13:26, Jan Lehnardt <jan@apache.org> wrote:
> 
>> 
>>> On 12 Oct 2015, at 13:15, Dale Harvey <dale@arandomurl.com> wrote:
>>> 
>>>> - instead of fixed test db names like `test_suite_db` that are used in
>>> almost every test,
>>>> we now generate a random database name for each test. The JS tests
>>> expected db
>>>> deletion to be synchronous, and that’s no longer the case in 2.0.
>> Instead
>>> of
>>>> error-prone polling to wait for db deletion, we just use new db for
>> every
>>> test.
>>>> An added benefit now is that (some) tests can run in parallel.
>>> 
>>> Little confused by this, PouchDB uses consistent names in its test suite
>>> (we had random
>>> ones for a while and moved away), we very much depend on the database
>> being
>>> deleted
>>> on the request returning and dont seem to have any problems with it
>> against
>>> master.
>> 
>> Does the PouchDB test suite runs against the 5986 port, or with an n=1
>> cluster?
>> This is with the default n=3 cluster against 5986.
>> 
>> Best
>> Jan
>> --
>> 
>>> 
>>> On 12 October 2015 at 12:21, Jan Lehnardt <jan@apache.org> wrote:
>>> 
>>>> 
>>>>> On 11 Oct 2015, at 18:53, Sebastian Rothbucher <
>>>> sebastianrothbucher@googlemail.com> wrote:
>>>>> 
>>>>> Hi Jan,
>>>>> 
>>>>> I transferred some of my findings to your branch - pls. find a PR for
>> the
>>>>> first few tests submitted 2 your branch. I put in TODOs where either
>>>> there
>>>>> is most probably a bug (filed COUCHDB-2852 and COUCHDB-2853) or we have
>>>> to
>>>>> do some more work (like making config available).
>>>> 
>>>> Excellent, thanks! :)
>>>> 
>>>> 
>>>>> Will from the Pouch team had the great idea of using -n 1 to mitigate
>>>>> Heisenbugs due to distribution. I adopted it and it did work quite
>> well.
>>>> It
>>>>> might make sense to put this into the dev/run call as a general rule,
>>>> esp.
>>>>> when checking for 201 (VS 202) status codes. Probably you know more on
>>>>> that.
>>>> 
>>>> We should carefully evaluate this :)
>>>> 
>>>> 
>>>>> Anyways: I think we have to keep using _config on the local port as
>> many
>>>>> tests (like auth tests) depend on changed configurations. BTW: w/ -n
1
>>>> it's
>>>>> only one call, but calling 3 _config(s) migt also work.
>>>>> Hopefully (I just arrived at some test w/ that recently) we can spare
>>>>> ourselves the _compact thing. I think we'd have to compact every single
>>>>> shard via local port, right? for the first tests, we can go w/out as
it
>>>>> seems. Maybe we should stick to that for a while.
>>>> 
>>>> My concern with this is that as far as I care, the local port is
>> irrelevant
>>>> to how CouchDB works. Of course it isn’t 100% irrelevant, but the
>> cluster
>>>> port is our publicised API for 2.0 and it’s the one that replaces the
>> 1.x
>>>> API for people upgrading. So having any tests that run on the local port
>>>> of 2.0 don’t help us in any way (that’s not to say it still useful to
>> have
>>>> tests against them for the moment).
>>>> 
>>>> Many of the _config-dependent tests substitute out the _users or
>>>> _replicator
>>>> databases, maybe we can find a workaround for those by just using the
>> stock
>>>> databases.
>>>> 
>>>> For the other tests, maybe we split them out into their own “test
>> suites”
>>>> that start their own test cluster in the configuration they need and
>> shut
>>>> it down afterwards.
>>>> 
>>>> Best
>>>> Jan
>>>> --
>>>> 
>>>> 
>>>>> 
>>>>> Looking fw 2 your thoughts
>>>>> 
>>>>> Best
>>>>>  Sebastian
>>>>> 
>>>>> P.S::  Maybe we can adopt some stuff from
>>>>> 
>>>> 
>> https://github.com/apache/couchdb/compare/master...sebastianrothbucher:clustertest
>>>>> 
>>>>> P.P.S.: Will keep working on it, just not in the next few days,
>>>>> unfortunately ;-(
>>>>> 
>>>>> On Sun, Oct 11, 2015 at 12:00 PM, Jan Lehnardt <jan@apache.org>
wrote:
>>>>> 
>>>>>> (I need your help deciding what to do with a bunch of tests, jump
to
>>>>>> “Discussion” below if you don’t care about the details before.)
>>>>>> 
>>>>>> * * *
>>>>>> 
>>>>>> Hi all,
>>>>>> 
>>>>>> I spent some more time getting the 1.x JavaScript tests running on
the
>>>>>> cluster port of 2.0. WIP here:
>>>> https://github.com/apache/couchdb/pull/354
>>>>>> — I’ve gotten a lot further, but there is still a lot to do and
>> everyone
>>>>>> here can help, I’ve posted instructions on how to do so further
down.
>> No
>>>>>> Erlang knowledge needed, this is ideal for folks who want to start
>> with
>>>>>> contributing to CouchDB.
>>>>>> 
>>>>>> 
>>>>>> Notable changes:
>>>>>> 
>>>>>> - instead of fixed test db names like `test_suite_db` that are used
in
>>>>>> almost every test, we now generate a random database name for each
>> test.
>>>>>> The JS tests expected db deletion to be synchronous, and that’s
no
>>>> longer
>>>>>> the case in 2.0. Instead of error-prone polling to wait for db
>>>> deletion, we
>>>>>> just use new db for every test. An added benefit now is that (some)
>>>> tests
>>>>>> can run in parallel.
>>>>>> 
>>>>>> - tests now clean up after themselves (incomplete): tests previously
>>>> only
>>>>>> collectively used three or four databases, and deleted and re-created
>>>> them
>>>>>> before each test was run. Now with the change of creating a new
>> database
>>>>>> per test, we also need to clean up databases at the end of a test,
>>>> because
>>>>>> otherwise we a) leave a lot of databases around and b) CouchDB runs
>> out
>>>> of
>>>>>> file descriptors during the test run. The cleanup isn’t 100% done
yet,
>>>> so
>>>>>> we should check for stray databases after JS test runs until we’ve
got
>>>> it
>>>>>> cleaned. One particular aspect here is when a test fails, it never
>>>> reaches
>>>>>> the cleanup code, so we might have to wrap everything into a try/catch
>>>>>> block.
>>>>>> 
>>>>>> - I disabled all tests that still fail and made them so they print
the
>>>>>> reason why they are failing, so that fixing them should be a lot
>> easier
>>>>>> now. Some of them print just “TODO”, their error cause is yet
to be
>>>>>> determined.
>>>>>> 
>>>>>> - couchdb.query() which previously executed temporary views, now
under
>>>> the
>>>>>> hood creates a design doc with a random name and queries that view.
>> This
>>>>>> fixes all tests relying on temp views without having to change them
>> all
>>>>>> individually. This is because 2.0 doesn’t have temp views anymore
>>>> (which is
>>>>>> a good thing).
>>>>>> 
>>>>>> - restartServer() is no longer used (mostly).
>>>>>> 
>>>>>> I already filed a few bugs that showed things missing in the cluster
>> API
>>>>>> that we’d expect to be there:
>>>>>> 
>>>>>> - https://issues.apache.org/jira/browse/COUCHDB-2849
>>>>>> - https://issues.apache.org/jira/browse/COUCHDB-2850
>>>>>> - https://issues.apache.org/jira/browse/COUCHDB-2851
>>>>>> 
>>>>>> Based on this I am ever more convinced we should get this test suite
>>>> work
>>>>>> against the 2.0 clustered API, because we’ll be finding some more
>>>>>> inconsistencies that we should fix before we invite beta testers.
>>>>>> 
>>>>>> 
>>>>>> ## Discussion:
>>>>>> 
>>>>>> We have a whole number of tests that use the test-internal function
>>>>>> run_on_modified_server. They use the _config API to test CouchDB
in
>>>>>> different configurations from the default one. All of these tests
now
>>>> throw
>>>>>> an error that _config isn’t available on the clustered API. How
should
>>>> we
>>>>>> handle them?
>>>>>> 
>>>>>> The tests are:
>>>>>> 
>>>>>> - erlang_views.js
>>>>>> - jsonp.js
>>>>>> - oauth_users_db.js
>>>>>> - oauth_users_db.js
>>>>>> - proxyauth.js
>>>>>> - reader_acl.js
>>>>>> - rewrite.js
>>>>>> - security_validation.js
>>>>>> - show_documents.js
>>>>>> - users_db.js
>>>>>> - view_errors.js
>>>>>> 
>>>>>> 
>>>>>> And a few tests aside from the proper compaction tests call _compact
>>>> which
>>>>>> is also not available on the cluster. What to do with them?
>>>>>> 
>>>>>> - recreate_doc.js
>>>>>> - and a few more that are currently disabled for other reasons, sorry
>> I
>>>>>> don’t have a full list at this point.
>>>>>> 
>>>>>> * * *
>>>>>> 
>>>>>> 
>>>>>> Bottom line: It’d be great if other folks here could join in the
fun.
>> No
>>>>>> Erlang needed, this is purely JavaScript and CouchDB HTTP API-work
:)
>>>>>> 
>>>>>> 
>>>>>> ## How to help:
>>>>>> 
>>>>>> 1. Check out https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip
>>>>>> 2. run `make clean && ./configure -c --disable-docs --disable-fauxton
>> &&
>>>>>> make`
>>>>>> 3. `make javascript` runs the whole test suite
>>>>>> 
>>>>>> To run the whole test suite without waiting for rebar to confirm
that
>> no
>>>>>> Erlang code needs to be compiled, run `./test/javascript/run` for
the
>>>> whole
>>>>>> test suite, or `./test/javascript/run all_docs` to run an individual
>>>> test
>>>>>> (all_docs runs test/javascript/tests/all_docs.js, other file names
>> match
>>>>>> accordingly).
>>>>>> 
>>>>>> Feel free to send me PRs against the branch on
>>>>>> https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip, I’ll
merge
>>>>>> them, so the PR on apache/couchdb gets updated.
>>>>>> 
>>>>>> I’m looking forward to your contributions! :)
>>>>>> 
>>>>>> If you have any questions or have trouble getting started or just
>> would
>>>>>> like some hand-holding, let me know!
>>>>>> 
>>>>>> 
>>>>>> Best
>>>>>> Jan
>>>>>> --
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>>> --
>>>> Professional Support for Apache CouchDB:
>>>> http://www.neighbourhood.ie/couchdb-support/
>>>> 
>>>> 
>> 
>> --
>> Professional Support for Apache CouchDB:
>> http://www.neighbourhood.ie/couchdb-support/
>> 
>> 

-- 
Professional Support for Apache CouchDB:
http://www.neighbourhood.ie/couchdb-support/


Mime
View raw message