couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Lehnardt <...@apache.org>
Subject Re: The JavaScript Test Suite
Date Mon, 12 Oct 2015 11:33:23 GMT
Oops, hits end too early.


Okay -n 1 explains this
> On 12 Oct 2015, at 13:32, Jan Lehnardt <jan@apache.org> wrote:
> 
> 
>> On 12 Oct 2015, at 13:30, Dale Harvey <dale@arandomurl.com> wrote:
>> 
>> Started with
>> 
>> python dev/run -n 1 --with-admin-party-please &
>> 
>> and export COUCH_HOST='http://127.0.0.1:15984'
>> 
>> 
>> On 12 October 2015 at 13:26, Jan Lehnardt <jan@apache.org> wrote:
>> 
>>> 
>>>> On 12 Oct 2015, at 13:15, Dale Harvey <dale@arandomurl.com> wrote:
>>>> 
>>>>> - instead of fixed test db names like `test_suite_db` that are used in
>>>> almost every test,
>>>>> we now generate a random database name for each test. The JS tests
>>>> expected db
>>>>> deletion to be synchronous, and that’s no longer the case in 2.0.
>>> Instead
>>>> of
>>>>> error-prone polling to wait for db deletion, we just use new db for
>>> every
>>>> test.
>>>>> An added benefit now is that (some) tests can run in parallel.
>>>> 
>>>> Little confused by this, PouchDB uses consistent names in its test suite
>>>> (we had random
>>>> ones for a while and moved away), we very much depend on the database
>>> being
>>>> deleted
>>>> on the request returning and dont seem to have any problems with it
>>> against
>>>> master.
>>> 
>>> Does the PouchDB test suite runs against the 5986 port, or with an n=1
>>> cluster?
>>> This is with the default n=3 cluster against 5986.
>>> 
>>> Best
>>> Jan
>>> --
>>> 
>>>> 
>>>> On 12 October 2015 at 12:21, Jan Lehnardt <jan@apache.org> wrote:
>>>> 
>>>>> 
>>>>>> On 11 Oct 2015, at 18:53, Sebastian Rothbucher <
>>>>> sebastianrothbucher@googlemail.com> wrote:
>>>>>> 
>>>>>> Hi Jan,
>>>>>> 
>>>>>> I transferred some of my findings to your branch - pls. find a PR
for
>>> the
>>>>>> first few tests submitted 2 your branch. I put in TODOs where either
>>>>> there
>>>>>> is most probably a bug (filed COUCHDB-2852 and COUCHDB-2853) or we
have
>>>>> to
>>>>>> do some more work (like making config available).
>>>>> 
>>>>> Excellent, thanks! :)
>>>>> 
>>>>> 
>>>>>> Will from the Pouch team had the great idea of using -n 1 to mitigate
>>>>>> Heisenbugs due to distribution. I adopted it and it did work quite
>>> well.
>>>>> It
>>>>>> might make sense to put this into the dev/run call as a general rule,
>>>>> esp.
>>>>>> when checking for 201 (VS 202) status codes. Probably you know more
on
>>>>>> that.
>>>>> 
>>>>> We should carefully evaluate this :)
>>>>> 
>>>>> 
>>>>>> Anyways: I think we have to keep using _config on the local port
as
>>> many
>>>>>> tests (like auth tests) depend on changed configurations. BTW: w/
-n 1
>>>>> it's
>>>>>> only one call, but calling 3 _config(s) migt also work.
>>>>>> Hopefully (I just arrived at some test w/ that recently) we can spare
>>>>>> ourselves the _compact thing. I think we'd have to compact every
single
>>>>>> shard via local port, right? for the first tests, we can go w/out
as it
>>>>>> seems. Maybe we should stick to that for a while.
>>>>> 
>>>>> My concern with this is that as far as I care, the local port is
>>> irrelevant
>>>>> to how CouchDB works. Of course it isn’t 100% irrelevant, but the
>>> cluster
>>>>> port is our publicised API for 2.0 and it’s the one that replaces the
>>> 1.x
>>>>> API for people upgrading. So having any tests that run on the local port
>>>>> of 2.0 don’t help us in any way (that’s not to say it still useful
to
>>> have
>>>>> tests against them for the moment).
>>>>> 
>>>>> Many of the _config-dependent tests substitute out the _users or
>>>>> _replicator
>>>>> databases, maybe we can find a workaround for those by just using the
>>> stock
>>>>> databases.
>>>>> 
>>>>> For the other tests, maybe we split them out into their own “test
>>> suites”
>>>>> that start their own test cluster in the configuration they need and
>>> shut
>>>>> it down afterwards.
>>>>> 
>>>>> Best
>>>>> Jan
>>>>> --
>>>>> 
>>>>> 
>>>>>> 
>>>>>> Looking fw 2 your thoughts
>>>>>> 
>>>>>> Best
>>>>>> Sebastian
>>>>>> 
>>>>>> P.S::  Maybe we can adopt some stuff from
>>>>>> 
>>>>> 
>>> https://github.com/apache/couchdb/compare/master...sebastianrothbucher:clustertest
>>>>>> 
>>>>>> P.P.S.: Will keep working on it, just not in the next few days,
>>>>>> unfortunately ;-(
>>>>>> 
>>>>>> On Sun, Oct 11, 2015 at 12:00 PM, Jan Lehnardt <jan@apache.org>
wrote:
>>>>>> 
>>>>>>> (I need your help deciding what to do with a bunch of tests,
jump to
>>>>>>> “Discussion” below if you don’t care about the details
before.)
>>>>>>> 
>>>>>>> * * *
>>>>>>> 
>>>>>>> Hi all,
>>>>>>> 
>>>>>>> I spent some more time getting the 1.x JavaScript tests running
on the
>>>>>>> cluster port of 2.0. WIP here:
>>>>> https://github.com/apache/couchdb/pull/354
>>>>>>> — I’ve gotten a lot further, but there is still a lot to
do and
>>> everyone
>>>>>>> here can help, I’ve posted instructions on how to do so further
down.
>>> No
>>>>>>> Erlang knowledge needed, this is ideal for folks who want to
start
>>> with
>>>>>>> contributing to CouchDB.
>>>>>>> 
>>>>>>> 
>>>>>>> Notable changes:
>>>>>>> 
>>>>>>> - instead of fixed test db names like `test_suite_db` that are
used in
>>>>>>> almost every test, we now generate a random database name for
each
>>> test.
>>>>>>> The JS tests expected db deletion to be synchronous, and that’s
no
>>>>> longer
>>>>>>> the case in 2.0. Instead of error-prone polling to wait for db
>>>>> deletion, we
>>>>>>> just use new db for every test. An added benefit now is that
(some)
>>>>> tests
>>>>>>> can run in parallel.
>>>>>>> 
>>>>>>> - tests now clean up after themselves (incomplete): tests previously
>>>>> only
>>>>>>> collectively used three or four databases, and deleted and re-created
>>>>> them
>>>>>>> before each test was run. Now with the change of creating a new
>>> database
>>>>>>> per test, we also need to clean up databases at the end of a
test,
>>>>> because
>>>>>>> otherwise we a) leave a lot of databases around and b) CouchDB
runs
>>> out
>>>>> of
>>>>>>> file descriptors during the test run. The cleanup isn’t 100%
done yet,
>>>>> so
>>>>>>> we should check for stray databases after JS test runs until
we’ve got
>>>>> it
>>>>>>> cleaned. One particular aspect here is when a test fails, it
never
>>>>> reaches
>>>>>>> the cleanup code, so we might have to wrap everything into a
try/catch
>>>>>>> block.
>>>>>>> 
>>>>>>> - I disabled all tests that still fail and made them so they
print the
>>>>>>> reason why they are failing, so that fixing them should be a
lot
>>> easier
>>>>>>> now. Some of them print just “TODO”, their error cause is
yet to be
>>>>>>> determined.
>>>>>>> 
>>>>>>> - couchdb.query() which previously executed temporary views,
now under
>>>>> the
>>>>>>> hood creates a design doc with a random name and queries that
view.
>>> This
>>>>>>> fixes all tests relying on temp views without having to change
them
>>> all
>>>>>>> individually. This is because 2.0 doesn’t have temp views anymore
>>>>> (which is
>>>>>>> a good thing).
>>>>>>> 
>>>>>>> - restartServer() is no longer used (mostly).
>>>>>>> 
>>>>>>> I already filed a few bugs that showed things missing in the
cluster
>>> API
>>>>>>> that we’d expect to be there:
>>>>>>> 
>>>>>>> - https://issues.apache.org/jira/browse/COUCHDB-2849
>>>>>>> - https://issues.apache.org/jira/browse/COUCHDB-2850
>>>>>>> - https://issues.apache.org/jira/browse/COUCHDB-2851
>>>>>>> 
>>>>>>> Based on this I am ever more convinced we should get this test
suite
>>>>> work
>>>>>>> against the 2.0 clustered API, because we’ll be finding some
more
>>>>>>> inconsistencies that we should fix before we invite beta testers.
>>>>>>> 
>>>>>>> 
>>>>>>> ## Discussion:
>>>>>>> 
>>>>>>> We have a whole number of tests that use the test-internal function
>>>>>>> run_on_modified_server. They use the _config API to test CouchDB
in
>>>>>>> different configurations from the default one. All of these tests
now
>>>>> throw
>>>>>>> an error that _config isn’t available on the clustered API.
How should
>>>>> we
>>>>>>> handle them?
>>>>>>> 
>>>>>>> The tests are:
>>>>>>> 
>>>>>>> - erlang_views.js
>>>>>>> - jsonp.js
>>>>>>> - oauth_users_db.js
>>>>>>> - oauth_users_db.js
>>>>>>> - proxyauth.js
>>>>>>> - reader_acl.js
>>>>>>> - rewrite.js
>>>>>>> - security_validation.js
>>>>>>> - show_documents.js
>>>>>>> - users_db.js
>>>>>>> - view_errors.js
>>>>>>> 
>>>>>>> 
>>>>>>> And a few tests aside from the proper compaction tests call _compact
>>>>> which
>>>>>>> is also not available on the cluster. What to do with them?
>>>>>>> 
>>>>>>> - recreate_doc.js
>>>>>>> - and a few more that are currently disabled for other reasons,
sorry
>>> I
>>>>>>> don’t have a full list at this point.
>>>>>>> 
>>>>>>> * * *
>>>>>>> 
>>>>>>> 
>>>>>>> Bottom line: It’d be great if other folks here could join in
the fun.
>>> No
>>>>>>> Erlang needed, this is purely JavaScript and CouchDB HTTP API-work
:)
>>>>>>> 
>>>>>>> 
>>>>>>> ## How to help:
>>>>>>> 
>>>>>>> 1. Check out https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip
>>>>>>> 2. run `make clean && ./configure -c --disable-docs --disable-fauxton
>>> &&
>>>>>>> make`
>>>>>>> 3. `make javascript` runs the whole test suite
>>>>>>> 
>>>>>>> To run the whole test suite without waiting for rebar to confirm
that
>>> no
>>>>>>> Erlang code needs to be compiled, run `./test/javascript/run`
for the
>>>>> whole
>>>>>>> test suite, or `./test/javascript/run all_docs` to run an individual
>>>>> test
>>>>>>> (all_docs runs test/javascript/tests/all_docs.js, other file
names
>>> match
>>>>>>> accordingly).
>>>>>>> 
>>>>>>> Feel free to send me PRs against the branch on
>>>>>>> https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip, I’ll
merge
>>>>>>> them, so the PR on apache/couchdb gets updated.
>>>>>>> 
>>>>>>> I’m looking forward to your contributions! :)
>>>>>>> 
>>>>>>> If you have any questions or have trouble getting started or
just
>>> would
>>>>>>> like some hand-holding, let me know!
>>>>>>> 
>>>>>>> 
>>>>>>> Best
>>>>>>> Jan
>>>>>>> --
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> --
>>>>> Professional Support for Apache CouchDB:
>>>>> http://www.neighbourhood.ie/couchdb-support/
>>>>> 
>>>>> 
>>> 
>>> --
>>> Professional Support for Apache CouchDB:
>>> http://www.neighbourhood.ie/couchdb-support/
>>> 
>>> 
> 
> -- 
> Professional Support for Apache CouchDB:
> http://www.neighbourhood.ie/couchdb-support/
> 

-- 
Professional Support for Apache CouchDB:
http://www.neighbourhood.ie/couchdb-support/


Mime
View raw message