From dev-return-48593-archive-asf-public=cust-asf.ponee.io@couchdb.apache.org Wed May 22 20:16:39 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 82EEB180651 for ; Wed, 22 May 2019 22:16:39 +0200 (CEST) Received: (qmail 63884 invoked by uid 500); 22 May 2019 20:16:38 -0000 Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@couchdb.apache.org Delivered-To: mailing list dev@couchdb.apache.org Received: (qmail 63873 invoked by uid 99); 22 May 2019 20:16:38 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 May 2019 20:16:38 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 8283DC2DDC for ; Wed, 22 May 2019 20:16:37 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.973 X-Spam-Level: X-Spam-Status: No, score=0.973 tagged_above=-999 required=6.31 tests=[SPF_HELO_NONE=0.001, SPF_SOFTFAIL=0.972] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id awrNx9tHdNYg for ; Wed, 22 May 2019 20:16:34 +0000 (UTC) Received: from smtp.justsomehost.net (smtp.justsomehost.net [204.11.51.157]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 6AA545FB19 for ; Wed, 22 May 2019 20:16:33 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp.justsomehost.net (Postfix) with ESMTP id C6F8658209C for ; Wed, 22 May 2019 16:16:24 -0400 (EDT) Received: from smtp.justsomehost.net ([127.0.0.1]) by localhost (smtp.justsomehost.net [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id i_9xqUjBUTit for ; Wed, 22 May 2019 16:16:24 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by smtp.justsomehost.net (Postfix) with ESMTP id 19A165820A1 for ; Wed, 22 May 2019 16:16:24 -0400 (EDT) X-Virus-Scanned: amavisd-new at smtp.justsomehost.net Received: from smtp.justsomehost.net ([127.0.0.1]) by localhost (smtp.justsomehost.net [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id uNuAgGgf5r8X for ; Wed, 22 May 2019 16:16:24 -0400 (EDT) Received: from [192.168.1.14] (toroon0560w-lp140-05-70-29-85-154.dsl.bell.ca [70.29.85.154]) by smtp.justsomehost.net (Postfix) with ESMTPSA id EB1A658209C for ; Wed, 22 May 2019 16:16:23 -0400 (EDT) Subject: Re: Use ExUnit to write unit tests. To: dev@couchdb.apache.org References: From: Joan Touzet Organization: Apache Software Foundation Message-ID: <17e5dcd3-6a5e-f0b9-9122-ae2eb221a5bf@apache.org> Date: Wed, 22 May 2019 16:16:18 -0400 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable Hi Ilya, thanks for starting this thread. Comments inline. On 2019-05-22 14:42, Ilya Khlopotov wrote: > The eunit testing framework is very hard to maintain. In particular, it= has the following problems: > - the process structure is designed in such a way that failure in setup= or teardown of one test affects the execution environment of subsequent = tests. Which makes it really hard to locate the place where the problem i= s coming from. I've personally experienced this a lot when reviewing failed logfiles, trying to find the *first* failure where things go wrong. It's a huge problem. > - inline test in the same module as the functions it tests might be ski= pped > - incorrect usage of ?assert vs ?_assert is not detectable since it mak= es tests pass=20 > - there is a weird (and hard to debug) interaction when used in combina= tion with meck=20 > - https://github.com/eproxus/meck/issues/133#issuecomment-113189678 > - https://github.com/eproxus/meck/issues/61 > - meck:unload() must be used instead of meck:unload(Module) Eep! I wasn't aware of this one. That's ugly. > - teardown is not always run, which affects all subsequent tests Have first-hand experienced this one too. > - grouping of tests is tricky > - it is hard to group tests so individual tests have meaningful descrip= tions >=20 > We believe that with ExUnit we wouldn't have these problems: Who's "we"? > - on_exit function is reliable in ExUnit > - it is easy to group tests using `describe` directive > - code-generation is trivial, which makes it is possible to generate te= sts from formal spec (if/when we have one) Can you address the timeout question w.r.t. EUnit that I raised elsewhere for cross-platform compatibility testing? I know that Peng ran into the same issues I did here and was looking into extending timeouts. Many of our tests suffer from failures where CI resources are slow and simply fail due to taking longer than expected. Does ExUnit have any additional support here? A suggestion was made (by Jay Doane, I believe, on IRC) that perhaps we simply remove all timeout=3D=3Dfailure logic (somehow?) and consider a timeout a hung test run, which would eventually fail the entire suite. This would ultimately lead to better deterministic testing, but we'd probably uncover quite a few bugs in the process (esp. against CouchDB <=3D 4.0). >=20 > Here are a few examples: >=20 > # Test adapters to test different interfaces using same test suite This is neat. I'd like someone else to comment whether this the approach you define will handle the polymorphic interfaces gracefully, or if the effort to parametrise/DRY out the tests will be more difficulty than simply maintaining 4 sets of tests. > # Using same test suite to compare new implementation of the same inter= face with the old one >=20 > Imagine that we are doing a major rewrite of a module which would imple= ment the same interface. *tries to imagine such a 'hypothetical' rewrite* :) > How do we compare both implementations return the same results for the = same input? > It is easy in Elixir, here is a sketch: Sounds interesting. I'd again like an analysis (from someone else) as to how straightforward this would be to implement. -Joan