couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Lehnardt <...@apache.org>
Subject Re: The JavaScript Test Suite
Date Mon, 12 Oct 2015 11:36:38 GMT

> On 12 Oct 2015, at 13:33, Jan Lehnardt <jan@apache.org> wrote:
> 
> Oops, hits end too early.
> 
> 
> Okay -n 1 explains this

And again, butterfingers!

…-n 1 explains this.

I’m afraid this hides potential issues because we are testing a condition
that is, while definitely common (presumably), not necessarily what still
a lot of people are running.

But maybe it’s okay for the tests like that, and we have to think about
cluster-aware integration tests in a separate step.

Would love to hear other opinions on this, maybe I’m just afraid for nothing,
or it’s justified, I just don’t know.

The (longer) experience PouchDB had here is definitely valuable, thanks Dale!

Best
Jan
--



>> On 12 Oct 2015, at 13:32, Jan Lehnardt <jan@apache.org> wrote:
>> 
>> 
>>> On 12 Oct 2015, at 13:30, Dale Harvey <dale@arandomurl.com> wrote:
>>> 
>>> Started with
>>> 
>>> python dev/run -n 1 --with-admin-party-please &
>>> 
>>> and export COUCH_HOST='http://127.0.0.1:15984'
>>> 
>>> 
>>> On 12 October 2015 at 13:26, Jan Lehnardt <jan@apache.org> wrote:
>>> 
>>>> 
>>>>> On 12 Oct 2015, at 13:15, Dale Harvey <dale@arandomurl.com> wrote:
>>>>> 
>>>>>> - instead of fixed test db names like `test_suite_db` that are used
in
>>>>> almost every test,
>>>>>> we now generate a random database name for each test. The JS tests
>>>>> expected db
>>>>>> deletion to be synchronous, and that’s no longer the case in 2.0.
>>>> Instead
>>>>> of
>>>>>> error-prone polling to wait for db deletion, we just use new db for
>>>> every
>>>>> test.
>>>>>> An added benefit now is that (some) tests can run in parallel.
>>>>> 
>>>>> Little confused by this, PouchDB uses consistent names in its test suite
>>>>> (we had random
>>>>> ones for a while and moved away), we very much depend on the database
>>>> being
>>>>> deleted
>>>>> on the request returning and dont seem to have any problems with it
>>>> against
>>>>> master.
>>>> 
>>>> Does the PouchDB test suite runs against the 5986 port, or with an n=1
>>>> cluster?
>>>> This is with the default n=3 cluster against 5986.
>>>> 
>>>> Best
>>>> Jan
>>>> --
>>>> 
>>>>> 
>>>>> On 12 October 2015 at 12:21, Jan Lehnardt <jan@apache.org> wrote:
>>>>> 
>>>>>> 
>>>>>>> On 11 Oct 2015, at 18:53, Sebastian Rothbucher <
>>>>>> sebastianrothbucher@googlemail.com> wrote:
>>>>>>> 
>>>>>>> Hi Jan,
>>>>>>> 
>>>>>>> I transferred some of my findings to your branch - pls. find
a PR for
>>>> the
>>>>>>> first few tests submitted 2 your branch. I put in TODOs where
either
>>>>>> there
>>>>>>> is most probably a bug (filed COUCHDB-2852 and COUCHDB-2853)
or we have
>>>>>> to
>>>>>>> do some more work (like making config available).
>>>>>> 
>>>>>> Excellent, thanks! :)
>>>>>> 
>>>>>> 
>>>>>>> Will from the Pouch team had the great idea of using -n 1 to
mitigate
>>>>>>> Heisenbugs due to distribution. I adopted it and it did work
quite
>>>> well.
>>>>>> It
>>>>>>> might make sense to put this into the dev/run call as a general
rule,
>>>>>> esp.
>>>>>>> when checking for 201 (VS 202) status codes. Probably you know
more on
>>>>>>> that.
>>>>>> 
>>>>>> We should carefully evaluate this :)
>>>>>> 
>>>>>> 
>>>>>>> Anyways: I think we have to keep using _config on the local port
as
>>>> many
>>>>>>> tests (like auth tests) depend on changed configurations. BTW:
w/ -n 1
>>>>>> it's
>>>>>>> only one call, but calling 3 _config(s) migt also work.
>>>>>>> Hopefully (I just arrived at some test w/ that recently) we can
spare
>>>>>>> ourselves the _compact thing. I think we'd have to compact every
single
>>>>>>> shard via local port, right? for the first tests, we can go w/out
as it
>>>>>>> seems. Maybe we should stick to that for a while.
>>>>>> 
>>>>>> My concern with this is that as far as I care, the local port is
>>>> irrelevant
>>>>>> to how CouchDB works. Of course it isn’t 100% irrelevant, but the
>>>> cluster
>>>>>> port is our publicised API for 2.0 and it’s the one that replaces
the
>>>> 1.x
>>>>>> API for people upgrading. So having any tests that run on the local
port
>>>>>> of 2.0 don’t help us in any way (that’s not to say it still useful
to
>>>> have
>>>>>> tests against them for the moment).
>>>>>> 
>>>>>> Many of the _config-dependent tests substitute out the _users or
>>>>>> _replicator
>>>>>> databases, maybe we can find a workaround for those by just using
the
>>>> stock
>>>>>> databases.
>>>>>> 
>>>>>> For the other tests, maybe we split them out into their own “test
>>>> suites”
>>>>>> that start their own test cluster in the configuration they need
and
>>>> shut
>>>>>> it down afterwards.
>>>>>> 
>>>>>> Best
>>>>>> Jan
>>>>>> --
>>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> Looking fw 2 your thoughts
>>>>>>> 
>>>>>>> Best
>>>>>>> Sebastian
>>>>>>> 
>>>>>>> P.S::  Maybe we can adopt some stuff from
>>>>>>> 
>>>>>> 
>>>> https://github.com/apache/couchdb/compare/master...sebastianrothbucher:clustertest
>>>>>>> 
>>>>>>> P.P.S.: Will keep working on it, just not in the next few days,
>>>>>>> unfortunately ;-(
>>>>>>> 
>>>>>>> On Sun, Oct 11, 2015 at 12:00 PM, Jan Lehnardt <jan@apache.org>
wrote:
>>>>>>> 
>>>>>>>> (I need your help deciding what to do with a bunch of tests,
jump to
>>>>>>>> “Discussion” below if you don’t care about the details
before.)
>>>>>>>> 
>>>>>>>> * * *
>>>>>>>> 
>>>>>>>> Hi all,
>>>>>>>> 
>>>>>>>> I spent some more time getting the 1.x JavaScript tests running
on the
>>>>>>>> cluster port of 2.0. WIP here:
>>>>>> https://github.com/apache/couchdb/pull/354
>>>>>>>> — I’ve gotten a lot further, but there is still a lot
to do and
>>>> everyone
>>>>>>>> here can help, I’ve posted instructions on how to do so
further down.
>>>> No
>>>>>>>> Erlang knowledge needed, this is ideal for folks who want
to start
>>>> with
>>>>>>>> contributing to CouchDB.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Notable changes:
>>>>>>>> 
>>>>>>>> - instead of fixed test db names like `test_suite_db` that
are used in
>>>>>>>> almost every test, we now generate a random database name
for each
>>>> test.
>>>>>>>> The JS tests expected db deletion to be synchronous, and
that’s no
>>>>>> longer
>>>>>>>> the case in 2.0. Instead of error-prone polling to wait for
db
>>>>>> deletion, we
>>>>>>>> just use new db for every test. An added benefit now is that
(some)
>>>>>> tests
>>>>>>>> can run in parallel.
>>>>>>>> 
>>>>>>>> - tests now clean up after themselves (incomplete): tests
previously
>>>>>> only
>>>>>>>> collectively used three or four databases, and deleted and
re-created
>>>>>> them
>>>>>>>> before each test was run. Now with the change of creating
a new
>>>> database
>>>>>>>> per test, we also need to clean up databases at the end of
a test,
>>>>>> because
>>>>>>>> otherwise we a) leave a lot of databases around and b) CouchDB
runs
>>>> out
>>>>>> of
>>>>>>>> file descriptors during the test run. The cleanup isn’t
100% done yet,
>>>>>> so
>>>>>>>> we should check for stray databases after JS test runs until
we’ve got
>>>>>> it
>>>>>>>> cleaned. One particular aspect here is when a test fails,
it never
>>>>>> reaches
>>>>>>>> the cleanup code, so we might have to wrap everything into
a try/catch
>>>>>>>> block.
>>>>>>>> 
>>>>>>>> - I disabled all tests that still fail and made them so they
print the
>>>>>>>> reason why they are failing, so that fixing them should be
a lot
>>>> easier
>>>>>>>> now. Some of them print just “TODO”, their error cause
is yet to be
>>>>>>>> determined.
>>>>>>>> 
>>>>>>>> - couchdb.query() which previously executed temporary views,
now under
>>>>>> the
>>>>>>>> hood creates a design doc with a random name and queries
that view.
>>>> This
>>>>>>>> fixes all tests relying on temp views without having to change
them
>>>> all
>>>>>>>> individually. This is because 2.0 doesn’t have temp views
anymore
>>>>>> (which is
>>>>>>>> a good thing).
>>>>>>>> 
>>>>>>>> - restartServer() is no longer used (mostly).
>>>>>>>> 
>>>>>>>> I already filed a few bugs that showed things missing in
the cluster
>>>> API
>>>>>>>> that we’d expect to be there:
>>>>>>>> 
>>>>>>>> - https://issues.apache.org/jira/browse/COUCHDB-2849
>>>>>>>> - https://issues.apache.org/jira/browse/COUCHDB-2850
>>>>>>>> - https://issues.apache.org/jira/browse/COUCHDB-2851
>>>>>>>> 
>>>>>>>> Based on this I am ever more convinced we should get this
test suite
>>>>>> work
>>>>>>>> against the 2.0 clustered API, because we’ll be finding
some more
>>>>>>>> inconsistencies that we should fix before we invite beta
testers.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> ## Discussion:
>>>>>>>> 
>>>>>>>> We have a whole number of tests that use the test-internal
function
>>>>>>>> run_on_modified_server. They use the _config API to test
CouchDB in
>>>>>>>> different configurations from the default one. All of these
tests now
>>>>>> throw
>>>>>>>> an error that _config isn’t available on the clustered
API. How should
>>>>>> we
>>>>>>>> handle them?
>>>>>>>> 
>>>>>>>> The tests are:
>>>>>>>> 
>>>>>>>> - erlang_views.js
>>>>>>>> - jsonp.js
>>>>>>>> - oauth_users_db.js
>>>>>>>> - oauth_users_db.js
>>>>>>>> - proxyauth.js
>>>>>>>> - reader_acl.js
>>>>>>>> - rewrite.js
>>>>>>>> - security_validation.js
>>>>>>>> - show_documents.js
>>>>>>>> - users_db.js
>>>>>>>> - view_errors.js
>>>>>>>> 
>>>>>>>> 
>>>>>>>> And a few tests aside from the proper compaction tests call
_compact
>>>>>> which
>>>>>>>> is also not available on the cluster. What to do with them?
>>>>>>>> 
>>>>>>>> - recreate_doc.js
>>>>>>>> - and a few more that are currently disabled for other reasons,
sorry
>>>> I
>>>>>>>> don’t have a full list at this point.
>>>>>>>> 
>>>>>>>> * * *
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Bottom line: It’d be great if other folks here could join
in the fun.
>>>> No
>>>>>>>> Erlang needed, this is purely JavaScript and CouchDB HTTP
API-work :)
>>>>>>>> 
>>>>>>>> 
>>>>>>>> ## How to help:
>>>>>>>> 
>>>>>>>> 1. Check out https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip
>>>>>>>> 2. run `make clean && ./configure -c --disable-docs
--disable-fauxton
>>>> &&
>>>>>>>> make`
>>>>>>>> 3. `make javascript` runs the whole test suite
>>>>>>>> 
>>>>>>>> To run the whole test suite without waiting for rebar to
confirm that
>>>> no
>>>>>>>> Erlang code needs to be compiled, run `./test/javascript/run`
for the
>>>>>> whole
>>>>>>>> test suite, or `./test/javascript/run all_docs` to run an
individual
>>>>>> test
>>>>>>>> (all_docs runs test/javascript/tests/all_docs.js, other file
names
>>>> match
>>>>>>>> accordingly).
>>>>>>>> 
>>>>>>>> Feel free to send me PRs against the branch on
>>>>>>>> https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip,
I’ll merge
>>>>>>>> them, so the PR on apache/couchdb gets updated.
>>>>>>>> 
>>>>>>>> I’m looking forward to your contributions! :)
>>>>>>>> 
>>>>>>>> If you have any questions or have trouble getting started
or just
>>>> would
>>>>>>>> like some hand-holding, let me know!
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Best
>>>>>>>> Jan
>>>>>>>> --
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Professional Support for Apache CouchDB:
>>>>>> http://www.neighbourhood.ie/couchdb-support/
>>>>>> 
>>>>>> 
>>>> 
>>>> --
>>>> Professional Support for Apache CouchDB:
>>>> http://www.neighbourhood.ie/couchdb-support/
>>>> 
>>>> 
>> 
>> -- 
>> Professional Support for Apache CouchDB:
>> http://www.neighbourhood.ie/couchdb-support/
>> 
> 
> -- 
> Professional Support for Apache CouchDB:
> http://www.neighbourhood.ie/couchdb-support/
> 

-- 
Professional Support for Apache CouchDB:
http://www.neighbourhood.ie/couchdb-support/


Mime
View raw message