Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
MIME-Version: 1.0
References: <CALpRrEiw=DWsAgkX3uBNecFrJPSodJjzxu1+gAV3WgKJrU-dGg@mail.gmail.com>
 <CABQ=58Q6me_eGob4zr8g52S6_c9Pek8gJnbHcyDeW-x1F6Tc+w@mail.gmail.com> <CAJ_m3YDr-t8XfFi5aJdoLU8Gf2A5WHKaxBTDc0uH_uuOoEkivg@mail.gmail.com>
In-Reply-To: <CAJ_m3YDr-t8XfFi5aJdoLU8Gf2A5WHKaxBTDc0uH_uuOoEkivg@mail.gmail.com>
From: Russell Branca <chewbranca@apache.org>
Date: Fri, 15 Dec 2017 17:42:56 +0000
Message-ID: <CALpRrEiTM902Gdjo6nCi54pji7v9N5NUfd+_cLP4gpZBYD=2mA@mail.gmail.com>
Subject: Re: [RFC] On the Testing of CouchDB
To: dev@couchdb.apache.org
Content-Type: multipart/alternative; boundary="f403043b2ae485e0e105606488c2"
archived-at: Fri, 15 Dec 2017 17:43:13 -0000

--f403043b2ae485e0e105606488c2
Content-Type: text/plain; charset="UTF-8"

Yeah just to reiterate what Paul said, the Elixir dev experience is really
nice and easy to get rolling with. I had no prior actual experience with
Elixir and I was able to get things rolling in a few hours.

RE Ben's question about diving in: please do! Just grab one of the unported
js suites and goto town. I've just been cherry-pick'ing things out of
Paul's branch and we can continue to do the same until we get this more
locked down. My goal with the porting is to keep chugging along and just
get it knocked out, as I really don't think it will be overly onerous to do
so. And if anyone else wants to jump in, there's still a fair number of
tests to port, just take your pick.

One other thing that needs work is figuring out how to hook all this into
"make check" and what not. I've mostly ignored that as this just points at
a CouchDB instance and can be run directly, but we'll need to sort that out
at some point.


-Russell

On Fri, Dec 15, 2017 at 9:03 AM Paul Davis <paul.joseph.davis@gmail.com>
wrote:

> Hello everybody!
>
> I figured I should probably go ahead and chime in seeing as I've also
> been playing around porting some of the tests in my free time between
> ops shifts the last couple weeks.
>
> My first impression was that it was ridiculously easy to get involved.
> On OS X at least, `brew install elixir` was enough to get a working
> elixir installed (however, if you use kerl or erln8 you'll want have
> to build an Erlang 20.x VM to use the brew package). I went from not
> having Elixir installed to a full port of uuids.js with the config tag
> logic written in about two hours one night. So far the Elixir docs and
> seem very well written and put together. I'd say the worst part of
> Elixir so far is that knowing Erlang I find myself searching for "How
> do I do this Erlang thing in Elixir?" Which isn't as bad as it sounds.
> The Elixir libraries have certainly had a considerable amount of
> thought put into them to make them easy to use and remember. I find it
> to be a lot like my experience when learning Python in that I may have
> to Google once and then its muscle memory. As opposed to Erlang's
> library where I'm constantly reading the lists manpage to remember
> argument orderings and whether I want search or find versions etc.
>
> Which I guess is a long way of saying I'm rather liking the Elixir
> development experience so far.
>
> That said, I'm currently about half way through porting replication.js
> tests to Elixir. For the most part its fairly straightforward. My
> current approach as we've done for the other modules is to do a direct
> port. Once that's finished we'll want to break up that huge module
> into a series of modules that share a lot of the utility functions.
> One of the nice things about moving to Elixir is that its got a full
> on development story rather than our current couchjs approach that
> prevents sharing code easily between subsets of tests.
>
> For Ben's question on diving in, I'd do just that. I'd say leave a
> note here about which module(s)? you're going to port so that we're
> not duplicating efforts and then its basically just a matter of
> getting Elixir installed. For that, here's a quick rundown on how I
> got that working:
>
> $ brew update
> $ brew install elixir
> $ # wait for all the things...
> $ iex # which fails cause I have an Erlang VM older than 20.0 as a default
> $ erln8 --fetch
> $ erln8 --build --tag=OTP-20.1.6 --id=20.1.6
> $ # wait while erln8 does its thing
> $ git clone https://github.com/apache/couchdb
> $ cd couchdb
> $ ./configure --disable-docs --disable-fauxton --with-curl
> $ make
> $ git checkout -b elixir-suite-davisp origin/elixir-suite # but use
> your own name
> $ cd elixir_suite # Russel promises to move this to test/elixir
> eventually... :)
> $ mix deps.get
> $ # For the moment, in another terminal, run ./dev/run -a adm:pass
> $ mix test --trace
> $ # For development you can also do this:
> $ mix test --trace test/module_i_am_working_on.exs
>
> For the time being, anyone that does any porting work, I'd just let
> Russel know and he can pull the changes into the main elixir-suite
> branch. For the initial work it might get a bit messy but we can
> always clean up after the fact if we decide this is a direction we'd
> like to go for real. To that end, I'd also make sure that we do a
> single .js -> .exs port per commit to try and make any future cleanup
> work easier.
>
> Also, even if people don't feel like doing any actual porting work I'd
> still be interested in hearing what its like for people to just run
> through their platform equivalent of the above steps. And even just
> initial impressions on toying around with Elixir. My only experience
> with Elixir prior to this was reading through their quick
> start/tutorial pages a couple of times to get a feeling for the syntax
> but hadn't actually even typed it into an editor till last week.
>
> And that's all I've got for now.
>
> On Thu, Dec 14, 2017 at 11:57 PM, Benjamin Anderson
> <banjiewen@apache.org> wrote:
> > Slick! This seems like it's coming together really nicely. Can't argue
> > with commits like "Prefer ?w=3 over hacky sleeps"[1] in any case.
> >
> >> I hope others have similar opinions after diving in!
> >
> > How should one dive in? Are you looking for others to help out with
> > the ports, or just thinking aspirationally about future regular
> > contributions to the test suite?
> >
> > --
> > b
> >
> > [1]:
> https://github.com/apache/couchdb/commit/5bce2d98a298c25b77d8dcda19deeedb494cc289
> >
> > On Thu, Dec 14, 2017 at 5:03 PM, Russell Branca <chewbranca@apache.org>
> wrote:
> >> Howdy folks!
> >>
> >> The testing of CouchDB is something that has seen focus and improvements
> >> for the last several years, for instance migrating the etap suite to
> eunit,
> >> and updating the JS suite to run against clusters in 2.x. There's still
> >> improvements to be made, and that was one of the topics of the CouchDB
> dev
> >> summit early in the year [1].
> >>
> >> Before we go further, I want to clarify some nomenclature. I'm by no
> means
> >> going to try and define unit testing vs integration testing vs quantum
> >> phase shift testing, but instead I want to focus on the distinction of
> >> where the testing takes place. Fundamentally, we have two places we test
> >> CouchDB: 1) at the Erlang VM level where we conduct assertions against
> >> module functions or process states; 2) at the HTTP level where we test
> the
> >> behavior of CouchDB at the user level API. This post focuses entirely on
> >> the latter; that's not to say the former doesn't also merit attention,
> just
> >> that the two are different enough that we can focus on them in
> isolation.
> >>
> >> So with that, let's chat about the current HTTP test suite in CouchDB.
> This
> >> is the "JS suite" I referred to above, which is a custom built test
> suite
> >> written in Javascript and executed in the aging SpiderMonkey. The JS
> suite
> >> has put in work for years, but it's showing it's age, and is a bit
> awkward
> >> to work with and improve. However, I think the biggest issue with the JS
> >> suite is that it's utilized far less than it should be, and folks seem
> to
> >> avoid extending it or adding additional tests to it. There's been
> >> discussion for years about replacing said suite, but the discussions
> >> invariably got blocked on the bike shed of whether to rewrite the suite
> in
> >> Javascript or Python. This thread provides a third option, with code!
> >>
> >> I started hacking on a replacement for the JS suite, this time written
> in
> >> Elixir. Overall I'm quite impressed with how it's come along, and have
> some
> >> good examples to show. This is basically an Elixir app that has an HTTP
> >> client and then runs a series of tests that conduct tests against the
> >> CouchDB HTTP API and make assertions therein.
> >>
> >> You can find the current code in [2], and a comparison of the changes in
> >> [3]. The core HTTP client is only a handful of lines of codes and works
> >> quite well [4]. The utility functions used across all tests are located
> in
> >> [5], and the tests themselves are in [6]. The existing test modules
> have a
> >> 1:1 correspondence with the associated JS suite test modules, and in
> >> general are as direct of a port as possible.
> >>
> >> The test modules ported in their entirety or most of the way are:
> >>
> >>   * all_docs.js
> >>   * basics.js
> >>   * config.js
> >>   * reduce.js
> >>   * rewrite.js
> >>   * uuids.js
> >>   * view_collation.js
> >>
> >> Paul has dove in and is responsible for a few of those test modules and
> >> he's almost completed porting the replication.js suite as well. We
> started
> >> with the hard ones first, so for the most part the rest of the ports
> should
> >> be fairly smooth sailing.
> >>
> >> Here's an example of a very basic test:
> >>
> >> ```erlang
> >> defmodule WelcomeTest do
> >>   use CouchTestCase
> >>
> >>   test "Welcome endpoint" do
> >>     assert Couch.get("/").body["couchdb"] == "Welcome", "Should say
> welcome"
> >>   end
> >>
> >> end
> >>
> >> ```
> >>
> >>
> >> As you can see, the `Couch` client is very simple HTTP client with
> >> easy HTTP verb based methods. Let's look at a more complicated test
> >> for asserting we can create documents in a database:
> >>
> >>
> >> ```erlang
> >>
> >>   @tag :with_db
> >>   test "Create a document and save it to the database", context do
> >>     resp = Couch.post("/#{context[:db_name]}", [body: %{:_id => "0",
> >> :a => 1, :b => 1}])
> >>     assert resp.status_code == 201, "Should be 201 created"
> >>     assert resp.body["id"], "Id should be present"
> >>     assert resp.body["rev"], "Rev should be present"
> >>
> >>     resp2 = Couch.get("/#{context[:db_name]}/#{resp.body["id"]}")
> >>     assert resp2.body["_id"] == resp.body["id"], "Ids should match"
> >>     assert resp2.body["_rev"] == resp.body["rev"], "Revs should match"
> >>   end
> >>
> >> ```
> >>
> >>
> >> This is fairly straightforward code to POST a new doc, make assertions
> >> on the response, and then fetch the doc to make sure everything
> >> matches up. What I really wanted to highlight here is the `@tag
> >> :with_db` decorator. We can easily add custom "tags" to the tests to
> >> simplify setup and teardown. That `:with_db` tag does two things, it
> >> dynamically generates a random database name, and then takes care of
> >> setup/teardown for creating and deleting said database for that
> >> particular test. This is really useful and has been very nice to work
> >> with so far. We also have tag functionality in place for executing a
> >> test with a particular set of config options:
> >>
> >>
> >> ```erlang
> >>
> >>   @tag config: [
> >>     {"uuids", "algorithm", "utc_random"}
> >>   ]
> >>   test "utc_random uuids are roughly random" do
> >>     resp = Couch.get("/_uuids", query: %{:count => 1000})
> >>     assert resp.status_code == 200
> >>     uuids = resp.body["uuids"]
> >>
> >>     assert String.length(Enum.at(uuids, 1)) == 32
> >>
> >>     # Assert no collisions
> >>     assert length(Enum.uniq(uuids)) == length(uuids)
> >>
> >>     # Assert rough ordering of UUIDs
> >>     u1 = String.slice(Enum.at(uuids, 1), 0..13)
> >>     u2 = String.slice(Enum.at(uuids, -1), 0..13)
> >>     assert u1 < u2
> >>   end
> >> ```
> >>
> >>
> >> The tag system really simplifies a lot of the standard auxiliary
> >> actions needed to conduct tests.
> >>
> >>
> >> To test out the suite, you'll need to spin up the dev server in one
> window with:
> >>
> >>
> >> ```
> >>
> >> ./dev/run --admin=adm:pass
> >>
> >> ```
> >>
> >>
> >> and then in another window go into the relevant CouchDB src directory
> and run:
> >>
> >>
> >> ```
> >>
> >> cd ~/src/couchdb/elixir_suite/
> >>
> >> mix deps.get
> >>
> >> mix test --trace
> >>
> >> ```
> >>
> >>
> >> The `--trace` flag makes the nice line item output per test, which I
> >> greatly prefer over a slew of periods. You can run an individual test
> >> with `mix test --trace tests/basics_test.exs`. I've pasted the output
> >> from running the basics suite at the bottom of this email so you can
> >> see what the real output looks like.
> >>
> >>
> >> Overall I'm quite impressed with the toolkit we've been able to put
> >> together in a short amount of time, and I propose we migrate fully to
> >> this test suite by porting all remaining JS suite tests and then
> >> removing the JS suite entirely. Given we've already ported most of the
> >> "hard suites", I think a full port is reasonable to do and just
> >> requires some leg work. Again, I'm impressed with how simple the
> >> tooling here is and how quickly we've been able to run with things,
> >> turns out the Elixir dev experience is actually quite nice! I hope
> >> others have similar opinions after diving in! Let me know what you
> >> think.
> >>
> >>
> >>
> >> -Russell
> >>
> >>
> >>
> >> [1] https://github.com/janl/couchdb-next/issues/39
> >> [2] https://github.com/apache/couchdb/tree/elixir-suite
> >> [3] https://github.com/apache/couchdb/compare/elixir-suite
> >> [4]
> >>
> https://github.com/apache/couchdb/blob/elixir-suite/elixir_suite/lib/couch.ex
> >> [5]
> >>
> https://github.com/apache/couchdb/blob/elixir-suite/elixir_suite/test/test_helper.exs
> >> [6]
> https://github.com/apache/couchdb/tree/elixir-suite/elixir_suite/test
> >>
> >>
> >> vagrant@contrib-jessie:~/src/couchdb/elixir_suite$ mix test --trace
> >> test/basics_test.exs
> >> Excluding tags: [pending: true]
> >>
> >> BasicsTest
> >>   * test Session contains adm context (66.8ms)
> >>   * test Creating a new DB with slashes should return Location header
> >> (COUCHDB-411) (85.8ms)
> >>   * test oops, the doc id got lost in code nirwana (82.1ms)
> >>   * test Welcome endpoint (7.6ms)
> >>   * test POST doc with an _id field isn't overwritten by uuid (102.7ms)
> >>   * test On restart, a request for creating an already existing db can
> >> not override (skipped)
> >>   * test Creating a new DB should return location header (118.7ms)
> >>   * test _bulk_docs POST error when body not an object (95.0ms)
> >>   * test Empty database should have zero docs (161.0ms)
> >>   * test _all_docs POST error when multi-get is not a {'key': [...]}
> >> structure (104.3ms)
> >>   * test Regression test for COUCHDB-954 (skipped)
> >>   * test DELETE'ing a non-existent doc should 404 (100.0ms)
> >>   * test Revs info status is good (127.3ms)
> >>   * test PUT on existing DB should return 412 instead of 500 (97.6ms)
> >>   * test Database should be in _all_dbs (117.7ms)
> >>   * test Check for invalid document members (122.4ms)
> >>   * test Can create several documents (213.0ms)
> >>   * test Make sure you can do a seq=true option (99.1ms)
> >>   * test PUT doc has a Location header (skipped)
> >>   * test Create a document and save it to the database (116.3ms)
> >>   * test Created database has appropriate db info name (99.7ms)
> >>   * test PUT error when body not an object (89.5ms)
> >>   * test Simple map functions (473.0ms)
> >>   * test POST doc response has a Location header (117.1ms)
> >>
> >> CouchTestCase
> >>
> >>
> >> Finished in 3.3 seconds
> >> 24 tests, 0 failures, 3 skipped
> >>
> >> Randomized with seed 936284
>

--f403043b2ae485e0e105606488c2--