Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
MIME-Version: 1.0
In-Reply-To: <CALpRrEiTM902Gdjo6nCi54pji7v9N5NUfd+_cLP4gpZBYD=2mA@mail.gmail.com>
References: <CALpRrEiw=DWsAgkX3uBNecFrJPSodJjzxu1+gAV3WgKJrU-dGg@mail.gmail.com>
 <CABQ=58Q6me_eGob4zr8g52S6_c9Pek8gJnbHcyDeW-x1F6Tc+w@mail.gmail.com>
 <CAJ_m3YDr-t8XfFi5aJdoLU8Gf2A5WHKaxBTDc0uH_uuOoEkivg@mail.gmail.com> <CALpRrEiTM902Gdjo6nCi54pji7v9N5NUfd+_cLP4gpZBYD=2mA@mail.gmail.com>
From: Paul Davis <paul.joseph.davis@gmail.com>
Date: Fri, 15 Dec 2017 11:45:38 -0600
Message-ID: <CAJ_m3YCVa8v-xu5pX_JK0horyKxg37kbJLjwWQzPfHNHFvQKPQ@mail.gmail.com>
Subject: Re: [RFC] On the Testing of CouchDB
To: dev@couchdb.apache.org
Content-Type: text/plain; charset="UTF-8"
archived-at: Fri, 15 Dec 2017 17:46:26 -0000

For `make check` it should be fairly straightforward to map the
current approach to it. I could probably knock that out fairly quickly
if you want me to give it a whirl.

On Fri, Dec 15, 2017 at 11:42 AM, Russell Branca <chewbranca@apache.org> wrote:
> Yeah just to reiterate what Paul said, the Elixir dev experience is really
> nice and easy to get rolling with. I had no prior actual experience with
> Elixir and I was able to get things rolling in a few hours.
>
> RE Ben's question about diving in: please do! Just grab one of the unported
> js suites and goto town. I've just been cherry-pick'ing things out of
> Paul's branch and we can continue to do the same until we get this more
> locked down. My goal with the porting is to keep chugging along and just
> get it knocked out, as I really don't think it will be overly onerous to do
> so. And if anyone else wants to jump in, there's still a fair number of
> tests to port, just take your pick.
>
> One other thing that needs work is figuring out how to hook all this into
> "make check" and what not. I've mostly ignored that as this just points at
> a CouchDB instance and can be run directly, but we'll need to sort that out
> at some point.
>
>
> -Russell
>
> On Fri, Dec 15, 2017 at 9:03 AM Paul Davis <paul.joseph.davis@gmail.com>
> wrote:
>
>> Hello everybody!
>>
>> I figured I should probably go ahead and chime in seeing as I've also
>> been playing around porting some of the tests in my free time between
>> ops shifts the last couple weeks.
>>
>> My first impression was that it was ridiculously easy to get involved.
>> On OS X at least, `brew install elixir` was enough to get a working
>> elixir installed (however, if you use kerl or erln8 you'll want have
>> to build an Erlang 20.x VM to use the brew package). I went from not
>> having Elixir installed to a full port of uuids.js with the config tag
>> logic written in about two hours one night. So far the Elixir docs and
>> seem very well written and put together. I'd say the worst part of
>> Elixir so far is that knowing Erlang I find myself searching for "How
>> do I do this Erlang thing in Elixir?" Which isn't as bad as it sounds.
>> The Elixir libraries have certainly had a considerable amount of
>> thought put into them to make them easy to use and remember. I find it
>> to be a lot like my experience when learning Python in that I may have
>> to Google once and then its muscle memory. As opposed to Erlang's
>> library where I'm constantly reading the lists manpage to remember
>> argument orderings and whether I want search or find versions etc.
>>
>> Which I guess is a long way of saying I'm rather liking the Elixir
>> development experience so far.
>>
>> That said, I'm currently about half way through porting replication.js
>> tests to Elixir. For the most part its fairly straightforward. My
>> current approach as we've done for the other modules is to do a direct
>> port. Once that's finished we'll want to break up that huge module
>> into a series of modules that share a lot of the utility functions.
>> One of the nice things about moving to Elixir is that its got a full
>> on development story rather than our current couchjs approach that
>> prevents sharing code easily between subsets of tests.
>>
>> For Ben's question on diving in, I'd do just that. I'd say leave a
>> note here about which module(s)? you're going to port so that we're
>> not duplicating efforts and then its basically just a matter of
>> getting Elixir installed. For that, here's a quick rundown on how I
>> got that working:
>>
>> $ brew update
>> $ brew install elixir
>> $ # wait for all the things...
>> $ iex # which fails cause I have an Erlang VM older than 20.0 as a default
>> $ erln8 --fetch
>> $ erln8 --build --tag=OTP-20.1.6 --id=20.1.6
>> $ # wait while erln8 does its thing
>> $ git clone https://github.com/apache/couchdb
>> $ cd couchdb
>> $ ./configure --disable-docs --disable-fauxton --with-curl
>> $ make
>> $ git checkout -b elixir-suite-davisp origin/elixir-suite # but use
>> your own name
>> $ cd elixir_suite # Russel promises to move this to test/elixir
>> eventually... :)
>> $ mix deps.get
>> $ # For the moment, in another terminal, run ./dev/run -a adm:pass
>> $ mix test --trace
>> $ # For development you can also do this:
>> $ mix test --trace test/module_i_am_working_on.exs
>>
>> For the time being, anyone that does any porting work, I'd just let
>> Russel know and he can pull the changes into the main elixir-suite
>> branch. For the initial work it might get a bit messy but we can
>> always clean up after the fact if we decide this is a direction we'd
>> like to go for real. To that end, I'd also make sure that we do a
>> single .js -> .exs port per commit to try and make any future cleanup
>> work easier.
>>
>> Also, even if people don't feel like doing any actual porting work I'd
>> still be interested in hearing what its like for people to just run
>> through their platform equivalent of the above steps. And even just
>> initial impressions on toying around with Elixir. My only experience
>> with Elixir prior to this was reading through their quick
>> start/tutorial pages a couple of times to get a feeling for the syntax
>> but hadn't actually even typed it into an editor till last week.
>>
>> And that's all I've got for now.
>>
>> On Thu, Dec 14, 2017 at 11:57 PM, Benjamin Anderson
>> <banjiewen@apache.org> wrote:
>> > Slick! This seems like it's coming together really nicely. Can't argue
>> > with commits like "Prefer ?w=3 over hacky sleeps"[1] in any case.
>> >
>> >> I hope others have similar opinions after diving in!
>> >
>> > How should one dive in? Are you looking for others to help out with
>> > the ports, or just thinking aspirationally about future regular
>> > contributions to the test suite?
>> >
>> > --
>> > b
>> >
>> > [1]:
>> https://github.com/apache/couchdb/commit/5bce2d98a298c25b77d8dcda19deeedb494cc289
>> >
>> > On Thu, Dec 14, 2017 at 5:03 PM, Russell Branca <chewbranca@apache.org>
>> wrote:
>> >> Howdy folks!
>> >>
>> >> The testing of CouchDB is something that has seen focus and improvements
>> >> for the last several years, for instance migrating the etap suite to
>> eunit,
>> >> and updating the JS suite to run against clusters in 2.x. There's still
>> >> improvements to be made, and that was one of the topics of the CouchDB
>> dev
>> >> summit early in the year [1].
>> >>
>> >> Before we go further, I want to clarify some nomenclature. I'm by no
>> means
>> >> going to try and define unit testing vs integration testing vs quantum
>> >> phase shift testing, but instead I want to focus on the distinction of
>> >> where the testing takes place. Fundamentally, we have two places we test
>> >> CouchDB: 1) at the Erlang VM level where we conduct assertions against
>> >> module functions or process states; 2) at the HTTP level where we test
>> the
>> >> behavior of CouchDB at the user level API. This post focuses entirely on
>> >> the latter; that's not to say the former doesn't also merit attention,
>> just
>> >> that the two are different enough that we can focus on them in
>> isolation.
>> >>
>> >> So with that, let's chat about the current HTTP test suite in CouchDB.
>> This
>> >> is the "JS suite" I referred to above, which is a custom built test
>> suite
>> >> written in Javascript and executed in the aging SpiderMonkey. The JS
>> suite
>> >> has put in work for years, but it's showing it's age, and is a bit
>> awkward
>> >> to work with and improve. However, I think the biggest issue with the JS
>> >> suite is that it's utilized far less than it should be, and folks seem
>> to
>> >> avoid extending it or adding additional tests to it. There's been
>> >> discussion for years about replacing said suite, but the discussions
>> >> invariably got blocked on the bike shed of whether to rewrite the suite
>> in
>> >> Javascript or Python. This thread provides a third option, with code!
>> >>
>> >> I started hacking on a replacement for the JS suite, this time written
>> in
>> >> Elixir. Overall I'm quite impressed with how it's come along, and have
>> some
>> >> good examples to show. This is basically an Elixir app that has an HTTP
>> >> client and then runs a series of tests that conduct tests against the
>> >> CouchDB HTTP API and make assertions therein.
>> >>
>> >> You can find the current code in [2], and a comparison of the changes in
>> >> [3]. The core HTTP client is only a handful of lines of codes and works
>> >> quite well [4]. The utility functions used across all tests are located
>> in
>> >> [5], and the tests themselves are in [6]. The existing test modules
>> have a
>> >> 1:1 correspondence with the associated JS suite test modules, and in
>> >> general are as direct of a port as possible.
>> >>
>> >> The test modules ported in their entirety or most of the way are:
>> >>
>> >>   * all_docs.js
>> >>   * basics.js
>> >>   * config.js
>> >>   * reduce.js
>> >>   * rewrite.js
>> >>   * uuids.js
>> >>   * view_collation.js
>> >>
>> >> Paul has dove in and is responsible for a few of those test modules and
>> >> he's almost completed porting the replication.js suite as well. We
>> started
>> >> with the hard ones first, so for the most part the rest of the ports
>> should
>> >> be fairly smooth sailing.
>> >>
>> >> Here's an example of a very basic test:
>> >>
>> >> ```erlang
>> >> defmodule WelcomeTest do
>> >>   use CouchTestCase
>> >>
>> >>   test "Welcome endpoint" do
>> >>     assert Couch.get("/").body["couchdb"] == "Welcome", "Should say
>> welcome"
>> >>   end
>> >>
>> >> end
>> >>
>> >> ```
>> >>
>> >>
>> >> As you can see, the `Couch` client is very simple HTTP client with
>> >> easy HTTP verb based methods. Let's look at a more complicated test
>> >> for asserting we can create documents in a database:
>> >>
>> >>
>> >> ```erlang
>> >>
>> >>   @tag :with_db
>> >>   test "Create a document and save it to the database", context do
>> >>     resp = Couch.post("/#{context[:db_name]}", [body: %{:_id => "0",
>> >> :a => 1, :b => 1}])
>> >>     assert resp.status_code == 201, "Should be 201 created"
>> >>     assert resp.body["id"], "Id should be present"
>> >>     assert resp.body["rev"], "Rev should be present"
>> >>
>> >>     resp2 = Couch.get("/#{context[:db_name]}/#{resp.body["id"]}")
>> >>     assert resp2.body["_id"] == resp.body["id"], "Ids should match"
>> >>     assert resp2.body["_rev"] == resp.body["rev"], "Revs should match"
>> >>   end
>> >>
>> >> ```
>> >>
>> >>
>> >> This is fairly straightforward code to POST a new doc, make assertions
>> >> on the response, and then fetch the doc to make sure everything
>> >> matches up. What I really wanted to highlight here is the `@tag
>> >> :with_db` decorator. We can easily add custom "tags" to the tests to
>> >> simplify setup and teardown. That `:with_db` tag does two things, it
>> >> dynamically generates a random database name, and then takes care of
>> >> setup/teardown for creating and deleting said database for that
>> >> particular test. This is really useful and has been very nice to work
>> >> with so far. We also have tag functionality in place for executing a
>> >> test with a particular set of config options:
>> >>
>> >>
>> >> ```erlang
>> >>
>> >>   @tag config: [
>> >>     {"uuids", "algorithm", "utc_random"}
>> >>   ]
>> >>   test "utc_random uuids are roughly random" do
>> >>     resp = Couch.get("/_uuids", query: %{:count => 1000})
>> >>     assert resp.status_code == 200
>> >>     uuids = resp.body["uuids"]
>> >>
>> >>     assert String.length(Enum.at(uuids, 1)) == 32
>> >>
>> >>     # Assert no collisions
>> >>     assert length(Enum.uniq(uuids)) == length(uuids)
>> >>
>> >>     # Assert rough ordering of UUIDs
>> >>     u1 = String.slice(Enum.at(uuids, 1), 0..13)
>> >>     u2 = String.slice(Enum.at(uuids, -1), 0..13)
>> >>     assert u1 < u2
>> >>   end
>> >> ```
>> >>
>> >>
>> >> The tag system really simplifies a lot of the standard auxiliary
>> >> actions needed to conduct tests.
>> >>
>> >>
>> >> To test out the suite, you'll need to spin up the dev server in one
>> window with:
>> >>
>> >>
>> >> ```
>> >>
>> >> ./dev/run --admin=adm:pass
>> >>
>> >> ```
>> >>
>> >>
>> >> and then in another window go into the relevant CouchDB src directory
>> and run:
>> >>
>> >>
>> >> ```
>> >>
>> >> cd ~/src/couchdb/elixir_suite/
>> >>
>> >> mix deps.get
>> >>
>> >> mix test --trace
>> >>
>> >> ```
>> >>
>> >>
>> >> The `--trace` flag makes the nice line item output per test, which I
>> >> greatly prefer over a slew of periods. You can run an individual test
>> >> with `mix test --trace tests/basics_test.exs`. I've pasted the output
>> >> from running the basics suite at the bottom of this email so you can
>> >> see what the real output looks like.
>> >>
>> >>
>> >> Overall I'm quite impressed with the toolkit we've been able to put
>> >> together in a short amount of time, and I propose we migrate fully to
>> >> this test suite by porting all remaining JS suite tests and then
>> >> removing the JS suite entirely. Given we've already ported most of the
>> >> "hard suites", I think a full port is reasonable to do and just
>> >> requires some leg work. Again, I'm impressed with how simple the
>> >> tooling here is and how quickly we've been able to run with things,
>> >> turns out the Elixir dev experience is actually quite nice! I hope
>> >> others have similar opinions after diving in! Let me know what you
>> >> think.
>> >>
>> >>
>> >>
>> >> -Russell
>> >>
>> >>
>> >>
>> >> [1] https://github.com/janl/couchdb-next/issues/39
>> >> [2] https://github.com/apache/couchdb/tree/elixir-suite
>> >> [3] https://github.com/apache/couchdb/compare/elixir-suite
>> >> [4]
>> >>
>> https://github.com/apache/couchdb/blob/elixir-suite/elixir_suite/lib/couch.ex
>> >> [5]
>> >>
>> https://github.com/apache/couchdb/blob/elixir-suite/elixir_suite/test/test_helper.exs
>> >> [6]
>> https://github.com/apache/couchdb/tree/elixir-suite/elixir_suite/test
>> >>
>> >>
>> >> vagrant@contrib-jessie:~/src/couchdb/elixir_suite$ mix test --trace
>> >> test/basics_test.exs
>> >> Excluding tags: [pending: true]
>> >>
>> >> BasicsTest
>> >>   * test Session contains adm context (66.8ms)
>> >>   * test Creating a new DB with slashes should return Location header
>> >> (COUCHDB-411) (85.8ms)
>> >>   * test oops, the doc id got lost in code nirwana (82.1ms)
>> >>   * test Welcome endpoint (7.6ms)
>> >>   * test POST doc with an _id field isn't overwritten by uuid (102.7ms)
>> >>   * test On restart, a request for creating an already existing db can
>> >> not override (skipped)
>> >>   * test Creating a new DB should return location header (118.7ms)
>> >>   * test _bulk_docs POST error when body not an object (95.0ms)
>> >>   * test Empty database should have zero docs (161.0ms)
>> >>   * test _all_docs POST error when multi-get is not a {'key': [...]}
>> >> structure (104.3ms)
>> >>   * test Regression test for COUCHDB-954 (skipped)
>> >>   * test DELETE'ing a non-existent doc should 404 (100.0ms)
>> >>   * test Revs info status is good (127.3ms)
>> >>   * test PUT on existing DB should return 412 instead of 500 (97.6ms)
>> >>   * test Database should be in _all_dbs (117.7ms)
>> >>   * test Check for invalid document members (122.4ms)
>> >>   * test Can create several documents (213.0ms)
>> >>   * test Make sure you can do a seq=true option (99.1ms)
>> >>   * test PUT doc has a Location header (skipped)
>> >>   * test Create a document and save it to the database (116.3ms)
>> >>   * test Created database has appropriate db info name (99.7ms)
>> >>   * test PUT error when body not an object (89.5ms)
>> >>   * test Simple map functions (473.0ms)
>> >>   * test POST doc response has a Location header (117.1ms)
>> >>
>> >> CouchTestCase
>> >>
>> >>
>> >> Finished in 3.3 seconds
>> >> 24 tests, 0 failures, 3 skipped
>> >>
>> >> Randomized with seed 936284
>>