Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
MIME-Version: 1.0
References: <CALpRrEiw=DWsAgkX3uBNecFrJPSodJjzxu1+gAV3WgKJrU-dGg@mail.gmail.com>
 <9C7ABFCC-73B6-400F-9560-4EFFB79ED021@apache.org>
In-Reply-To: <9C7ABFCC-73B6-400F-9560-4EFFB79ED021@apache.org>
From: Russell Branca <russell@chewbranca.com>
Date: Sat, 16 Dec 2017 18:12:37 +0000
Message-ID: <CALpRrEg8iZZrg0d1WxgUvJEoyJ+23-07F+GJBq5Jh31DTuwLeA@mail.gmail.com>
Subject: Re: [RFC] On the Testing of CouchDB
To: dev@couchdb.apache.org
Content-Type: multipart/alternative; boundary="001a113a1f168aebb105607910ea"
archived-at: Sat, 16 Dec 2017 18:13:04 -0000

--001a113a1f168aebb105607910ea
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hey Jan, thanks for the feedback. Handful of replies inline.

On Sat, Dec 16, 2017 at 3:18 AM Jan Lehnardt <jan@apache.org> wrote:

> Heyall,
>
> this is really cool. Im very happy to see progress on our test suites
> and this looks like a great way forward.
>
> Yeah, me too!


> A little bit more context on the JS suite and its runtime environment
> couchjs: couchjs was never meant to be a general purpose JS execution
> environment. We started using couchjs for the JS test suite because it
> was already available in CouchDB and we were using JS tests as opposed
> to Erlang tests because they were meant as initial documentation of
> CouchDBs API. That was before the wiki was useful and dedicated docs
> were nowhere in sight.
>
> We are even maintaining a custom, C-based HTTP client so we can use
> couchjs for HTTP tests. I=E2=80=99d be very happy to get rid of that code=
.
>
> Today, there are better JS runtimes that are easy to use, and the JS
> tests changed purpose from being documentation to being the source of
> validation for a lot of API quirks in CouchDB that are not encoded
> anywhere else outside of CouchDB. That=E2=80=99s why it is useful to keep=
 the
> knowledge inside the JS test suite around, but bring it forward to a
> more usable execution environment.
>
> * * *
>
> When we=E2=80=99ve discussed the JS test suite in the past, we talked abo=
ut
> joining forces with the PouchDB team, since they have a version of our
> JS tests running, with many good additions. Going with Elixir, we=E2=80=
=99re
> closing the door on that particular collaboration, but given that this
> has been discussed for 2+ years and not much happened, I=E2=80=99m okay w=
ith the
> Elixir suite. The pouchdb-server project will be able to use the Elixir
> suite just as we do.
>
> The one thing that would be nice here if it were easy to disable certain
> tests or suites that make no sense in the pouchdb-server environment, so
> they can easily integrate it in their CI.
>

This is actually well supported with the ExUnit engine from Elixir, and in
fact we're already basically doing this. The tag system we used for
creating dbs and setting configs is also utilized as a filtering system for
tests. You can see a handful of examples and options by running "mix help
test". In the link below you can see how we're currently skipping tests
tagged with "pending", and similarly, you can set module level tags in
addition to test specific tags. As long as we're diligent with setting test
tags, it will be trivial to run a specific subset of tests with something
like "mix test --exclude breaks-pouchdb" or some such.

https://github.com/apache/couchdb/blob/elixir-suite/test/elixir/test/test_h=
elper.exs#L1


> * * *
>
> One of the problems with the JS tests is that they predate testing best
> practices. The majority of tests are single functions with sometimes
> hundreds of assertions and changes in configuration and mock data. This
> is not very maintainable. The stats.js tests are split up more
> traditionally, but that effort was never extended to the other test
> files.
>
> I was happy to see that the Elixir port of the basics.js tests took a
> sensible approach breaking things up in the way a modern test suite
> would be done. Advantages include better isolation of specific behaviour
> tests, ease of understanding tests, simpler running of individual tests
> during development, more chances to structure and re-use mock data and
> other boilerplate code, all making any work with the tests more pleasant.
>
> It would be great if we could use this opportunity to apply this across
> all JS test files when we port them to Elixir. It means a little bit
> more work per test file, but I hope with a few more contributors and
> guidelines, this is an easily paralleliseable task, so individual burden
> can be minimised.
>
> Yeah I agree completely. The basics suite was fairly amenable to be being
broken up in that manner, however there are other suites that were not as
easy to break apart like that, for instance the all docs suite linked
below. My general approach so far has been to break things apart into
isolated tests whenever there's a simple and clear path to do so, and when
not just do as direct of a port of the tests as possible. IMO we should do
a first round of porting the tests over directly, and then once we've got
them all in Elixir we should do some refactoring to simplify and
restructure things as appropriate.

https://github.com/apache/couchdb/blob/elixir-suite/test/elixir/test/all_do=
cs_test.exs


> * * *
>
> I noticed that one of the reduce tests took 30+ seconds to run on my
> machine and I experimented with different cluster configuration values
> and to nobodys surprise, the default of q=3D8 is the main factor in view
> test execution speed. q=3D4 takes ~20s, q=3D2 ~10s and q=3D1 ~5s. I=E2=80=
=99m not
> suggesting we set q=3D1 for all tests since q>1 is a behaviour we would
> want to test as well, but maybe we can set q=3D2 when running the test
> suite(s) for the time being. Shaving 25s off of a single test will get
> us a long way with all tests ported. What do others think?
>

I'm not sure what's going on here, although Paul ran into similar
situations where tests would run on my machine in under a second but take
significantly longer on his machine. I think we should investigate this a
bit more before taking any major action, but for a lot of tests running
with a lower Q value would probably be fine. I think it's important we run
with Q >=3D 2 so we at least run through the standard code paths of having =
to
merge over multiple shard ranges for aggregate operations. Although that
said, it could also be worthwhile to run tests with Q=3D1 to exercise the n=
on
merge code paths as well.


>
> @Nick: you introduced the -C / --config-overrides option to dev/run,
> but I could never figure out how to apply it. That would be the easiest
> place to make the cluster config change for the Elixir tests.
>
> `-c q=3D2` makes the nodes fail to start and I=E2=80=99m not sure how in =
this case,
> the `[cluster]` section is meant.
>
> * * *
>
> Russell, Paul: do you think it is worth reaching out to the Elixir
> community and ask if they are interested in helping out a little? If
> you think its too early, we can wait with this.
>

Yeah I think that's a great idea and could be an excellent opportunity to
get more folks involved in things. I do think we should probably figure out
our branch/dev strategy prior to doing that, as me cherry-picking commits
will eventually become onerous given sufficient branches.


> * * *
>
> Thanks again!
>
> Best
> Jan
> --
>
>
No problem, really happy to see folks excited about this new engine!


-Russell


>
>
> > On 15. Dec 2017, at 02:03, Russell Branca <chewbranca@apache.org> wrote=
:
> >
> > Howdy folks!
> >
> > The testing of CouchDB is something that has seen focus and improvement=
s
> > for the last several years, for instance migrating the etap suite to
> eunit,
> > and updating the JS suite to run against clusters in 2.x. There's still
> > improvements to be made, and that was one of the topics of the CouchDB
> dev
> > summit early in the year [1].
> >
> > Before we go further, I want to clarify some nomenclature. I'm by no
> means
> > going to try and define unit testing vs integration testing vs quantum
> > phase shift testing, but instead I want to focus on the distinction of
> > where the testing takes place. Fundamentally, we have two places we tes=
t
> > CouchDB: 1) at the Erlang VM level where we conduct assertions against
> > module functions or process states; 2) at the HTTP level where we test
> the
> > behavior of CouchDB at the user level API. This post focuses entirely o=
n
> > the latter; that's not to say the former doesn't also merit attention,
> just
> > that the two are different enough that we can focus on them in isolatio=
n.
> >
> > So with that, let's chat about the current HTTP test suite in CouchDB.
> This
> > is the "JS suite" I referred to above, which is a custom built test sui=
te
> > written in Javascript and executed in the aging SpiderMonkey. The JS
> suite
> > has put in work for years, but it's showing it's age, and is a bit
> awkward
> > to work with and improve. However, I think the biggest issue with the J=
S
> > suite is that it's utilized far less than it should be, and folks seem =
to
> > avoid extending it or adding additional tests to it. There's been
> > discussion for years about replacing said suite, but the discussions
> > invariably got blocked on the bike shed of whether to rewrite the suite
> in
> > Javascript or Python. This thread provides a third option, with code!
> >
> > I started hacking on a replacement for the JS suite, this time written =
in
> > Elixir. Overall I'm quite impressed with how it's come along, and have
> some
> > good examples to show. This is basically an Elixir app that has an HTTP
> > client and then runs a series of tests that conduct tests against the
> > CouchDB HTTP API and make assertions therein.
> >
> > You can find the current code in [2], and a comparison of the changes i=
n
> > [3]. The core HTTP client is only a handful of lines of codes and works
> > quite well [4]. The utility functions used across all tests are located
> in
> > [5], and the tests themselves are in [6]. The existing test modules hav=
e
> a
> > 1:1 correspondence with the associated JS suite test modules, and in
> > general are as direct of a port as possible.
> >
> > The test modules ported in their entirety or most of the way are:
> >
> >  * all_docs.js
> >  * basics.js
> >  * config.js
> >  * reduce.js
> >  * rewrite.js
> >  * uuids.js
> >  * view_collation.js
> >
> > Paul has dove in and is responsible for a few of those test modules and
> > he's almost completed porting the replication.js suite as well. We
> started
> > with the hard ones first, so for the most part the rest of the ports
> should
> > be fairly smooth sailing.
> >
> > Here's an example of a very basic test:
> >
> > ```erlang
> > defmodule WelcomeTest do
> >  use CouchTestCase
> >
> >  test "Welcome endpoint" do
> >    assert Couch.get("/").body["couchdb"] =3D=3D "Welcome", "Should say
> welcome"
> >  end
> >
> > end
> >
> > ```
> >
> >
> > As you can see, the `Couch` client is very simple HTTP client with
> > easy HTTP verb based methods. Let's look at a more complicated test
> > for asserting we can create documents in a database:
> >
> >
> > ```erlang
> >
> >  @tag :with_db
> >  test "Create a document and save it to the database", context do
> >    resp =3D Couch.post("/#{context[:db_name]}", [body: %{:_id =3D> "0",
> > :a =3D> 1, :b =3D> 1}])
> >    assert resp.status_code =3D=3D 201, "Should be 201 created"
> >    assert resp.body["id"], "Id should be present"
> >    assert resp.body["rev"], "Rev should be present"
> >
> >    resp2 =3D Couch.get("/#{context[:db_name]}/#{resp.body["id"]}")
> >    assert resp2.body["_id"] =3D=3D resp.body["id"], "Ids should match"
> >    assert resp2.body["_rev"] =3D=3D resp.body["rev"], "Revs should matc=
h"
> >  end
> >
> > ```
> >
> >
> > This is fairly straightforward code to POST a new doc, make assertions
> > on the response, and then fetch the doc to make sure everything
> > matches up. What I really wanted to highlight here is the `@tag
> > :with_db` decorator. We can easily add custom "tags" to the tests to
> > simplify setup and teardown. That `:with_db` tag does two things, it
> > dynamically generates a random database name, and then takes care of
> > setup/teardown for creating and deleting said database for that
> > particular test. This is really useful and has been very nice to work
> > with so far. We also have tag functionality in place for executing a
> > test with a particular set of config options:
> >
> >
> > ```erlang
> >
> >  @tag config: [
> >    {"uuids", "algorithm", "utc_random"}
> >  ]
> >  test "utc_random uuids are roughly random" do
> >    resp =3D Couch.get("/_uuids", query: %{:count =3D> 1000})
> >    assert resp.status_code =3D=3D 200
> >    uuids =3D resp.body["uuids"]
> >
> >    assert String.length(Enum.at(uuids, 1)) =3D=3D 32
> >
> >    # Assert no collisions
> >    assert length(Enum.uniq(uuids)) =3D=3D length(uuids)
> >
> >    # Assert rough ordering of UUIDs
> >    u1 =3D String.slice(Enum.at(uuids, 1), 0..13)
> >    u2 =3D String.slice(Enum.at(uuids, -1), 0..13)
> >    assert u1 < u2
> >  end
> > ```
> >
> >
> > The tag system really simplifies a lot of the standard auxiliary
> > actions needed to conduct tests.
> >
> >
> > To test out the suite, you'll need to spin up the dev server in one
> window with:
> >
> >
> > ```
> >
> > ./dev/run --admin=3Dadm:pass
> >
> > ```
> >
> >
> > and then in another window go into the relevant CouchDB src directory
> and run:
> >
> >
> > ```
> >
> > cd ~/src/couchdb/elixir_suite/
> >
> > mix deps.get
> >
> > mix test --trace
> >
> > ```
> >
> >
> > The `--trace` flag makes the nice line item output per test, which I
> > greatly prefer over a slew of periods. You can run an individual test
> > with `mix test --trace tests/basics_test.exs`. I've pasted the output
> > from running the basics suite at the bottom of this email so you can
> > see what the real output looks like.
> >
> >
> > Overall I'm quite impressed with the toolkit we've been able to put
> > together in a short amount of time, and I propose we migrate fully to
> > this test suite by porting all remaining JS suite tests and then
> > removing the JS suite entirely. Given we've already ported most of the
> > "hard suites", I think a full port is reasonable to do and just
> > requires some leg work. Again, I'm impressed with how simple the
> > tooling here is and how quickly we've been able to run with things,
> > turns out the Elixir dev experience is actually quite nice! I hope
> > others have similar opinions after diving in! Let me know what you
> > think.
> >
> >
> >
> > -Russell
> >
> >
> >
> > [1] https://github.com/janl/couchdb-next/issues/39
> > [2] https://github.com/apache/couchdb/tree/elixir-suite
> > [3] https://github.com/apache/couchdb/compare/elixir-suite
> > [4]
> >
> https://github.com/apache/couchdb/blob/elixir-suite/elixir_suite/lib/couc=
h.ex
> > [5]
> >
> https://github.com/apache/couchdb/blob/elixir-suite/elixir_suite/test/tes=
t_helper.exs
> > [6]
> https://github.com/apache/couchdb/tree/elixir-suite/elixir_suite/test
> >
> >
> > vagrant@contrib-jessie:~/src/couchdb/elixir_suite$ mix test --trace
> > test/basics_test.exs
> > Excluding tags: [pending: true]
> >
> > BasicsTest
> >  * test Session contains adm context (66.8ms)
> >  * test Creating a new DB with slashes should return Location header
> > (COUCHDB-411) (85.8ms)
> >  * test oops, the doc id got lost in code nirwana (82.1ms)
> >  * test Welcome endpoint (7.6ms)
> >  * test POST doc with an _id field isn't overwritten by uuid (102.7ms)
> >  * test On restart, a request for creating an already existing db can
> > not override (skipped)
> >  * test Creating a new DB should return location header (118.7ms)
> >  * test _bulk_docs POST error when body not an object (95.0ms)
> >  * test Empty database should have zero docs (161.0ms)
> >  * test _all_docs POST error when multi-get is not a {'key': [...]}
> > structure (104.3ms)
> >  * test Regression test for COUCHDB-954 (skipped)
> >  * test DELETE'ing a non-existent doc should 404 (100.0ms)
> >  * test Revs info status is good (127.3ms)
> >  * test PUT on existing DB should return 412 instead of 500 (97.6ms)
> >  * test Database should be in _all_dbs (117.7ms)
> >  * test Check for invalid document members (122.4ms)
> >  * test Can create several documents (213.0ms)
> >  * test Make sure you can do a seq=3Dtrue option (99.1ms)
> >  * test PUT doc has a Location header (skipped)
> >  * test Create a document and save it to the database (116.3ms)
> >  * test Created database has appropriate db info name (99.7ms)
> >  * test PUT error when body not an object (89.5ms)
> >  * test Simple map functions (473.0ms)
> >  * test POST doc response has a Location header (117.1ms)
> >
> > CouchTestCase
> >
> >
> > Finished in 3.3 seconds
> > 24 tests, 0 failures, 3 skipped
> >
> > Randomized with seed 936284
>
> --
> Professional Support for Apache CouchDB:
> https://neighbourhood.ie/couchdb-support/
>
>

--001a113a1f168aebb105607910ea--