From dev-return-30313-archive-asf-public=cust-asf.ponee.io@geode.apache.org Wed Nov 21 19:49:10 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id B894F180675 for ; Wed, 21 Nov 2018 19:49:09 +0100 (CET) Received: (qmail 40582 invoked by uid 500); 21 Nov 2018 18:49:08 -0000 Mailing-List: contact dev-help@geode.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@geode.apache.org Delivered-To: mailing list dev@geode.apache.org Received: (qmail 40563 invoked by uid 99); 21 Nov 2018 18:49:08 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Nov 2018 18:49:08 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 96A97C21C3 for ; Wed, 21 Nov 2018 18:49:07 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.3 X-Spam-Level: * X-Spam-Status: No, score=1.3 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id zwaa9ICerWnr for ; Wed, 21 Nov 2018 18:49:05 +0000 (UTC) Received: from mx0b-00296801.pphosted.com (mx0b-00296801.pphosted.com [148.163.153.148]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 5C1CC60EB5 for ; Wed, 21 Nov 2018 18:41:33 +0000 (UTC) Received: from pps.filterd (m0114585.ppops.net [127.0.0.1]) by mx0b-00296801.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id wALIfDJG012927 for ; Wed, 21 Nov 2018 18:41:32 GMT Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by mx0b-00296801.pphosted.com with ESMTP id 2ntb56kx8s-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 21 Nov 2018 18:41:31 +0000 Received: by mail-pl1-f197.google.com with SMTP id o23so10258078pll.0 for ; Wed, 21 Nov 2018 10:41:31 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:mime-version:subject:date:references:to :in-reply-to:message-id; bh=612Mm8XD10sqMXbWToyTnibOuMw7eIBeVc/p7ja5YH4=; b=oXHL5vV3vf0E/eM2lkP5Gn01YHS16zaGfEYsR4tHw5PQvmCprxPPNs7HqISmn5JKu6 FuyR2VsLJqvEzHnU3TNZ49LnrcEX/09vWR335tTrHvb0m9qBimTLomdMTmJ2Ogat1q6t KtOmJMYRp7hkMHQpXVqQojSb9abJQolMpZP1TYQ0pxeCXfoZ5dVn33FOcY666eRY82Xk cVeW0ZANKGmoYP430YWj8JZUWHe8y0QE587g+BckFAMTwdNnEMOVPB1x4EdfZaIffJvh C/P8gCz1mYnJE2EkufwXlcpTe2iqKDh23ozdYfsdz0N0qOk7ZhUa2nGUAQx4alHDXOXZ T4WQ== X-Gm-Message-State: AA+aEWanPXE0YZxjc1YMK3Y8h7Xma9/bjEnOlarGrojqoMOw8MGn8wTj 0c0R5aQ8wqzViVzRWLoWBVrGZHaauSzIE/EkR3LvBbQEHxNomgOkPR6Q/L4o3s6XLUSZK2NXRRd 557alqwpxmczDS01XeqiHeW81w+9TKZfNctIAUjY= X-Received: by 2002:a17:902:bb98:: with SMTP id m24mr7793514pls.71.1542825689628; Wed, 21 Nov 2018 10:41:29 -0800 (PST) X-Google-Smtp-Source: AFSGD/WlSkuG8vWy+EHHXVRMG9ZYJpSyQbDumVmibfqyEbY2N0knzFFvy8GJKsdu6SPTQtINkoSUOg== X-Received: by 2002:a17:902:bb98:: with SMTP id m24mr7793495pls.71.1542825689130; Wed, 21 Nov 2018 10:41:29 -0800 (PST) Received: from [10.118.33.252] (50-203-225-134-static.hfc.comcastbusiness.net. [50.203.225.134]) by smtp.gmail.com with ESMTPSA id t13sm92572882pgr.42.2018.11.21.10.41.27 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 21 Nov 2018 10:41:28 -0800 (PST) From: Nabarun Nag Content-Type: multipart/alternative; boundary="Apple-Mail=_C13B5581-49DA-47F7-A327-756F9950B2D5" Mime-Version: 1.0 (Mac OS X Mail 12.1 \(3445.101.1\)) Subject: Re: Avoiding Unpredictability in Our DUnit Testing Rules Date: Wed, 21 Nov 2018 10:41:26 -0800 References: To: dev@geode.apache.org In-Reply-To: Message-Id: X-Mailer: Apple Mail (2.3445.101.1) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2018-11-21_09:,, signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1811210162 --Apple-Mail=_C13B5581-49DA-47F7-A327-756F9950B2D5 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Will doing this in the test, final Host host =3D Host.getHost(0); VM server1 =3D host.getVM(startingVersion, 0); be frowned upon, if I use the above over using @Rule. Regards Nabarun Nag > On Nov 21, 2018, at 10:36 AM, Robert Houghton = wrote: >=20 > Great find, Patrick. I hope this shakes out some of the test bugs! >=20 > On Wed, Nov 21, 2018, 10:34 Patrick Rhomberg =20 >> tl;dr: Use JUnit RuleChain. >> ---- >>=20 >> Hello all! >>=20 >> Several [1] of our test @Rule classes make use of the fact that our = DUnit >> VMs Host is statically accessible to affect every test VM. For = instance, >> the SharedCountersRule initializes a counter in every VM, and the >> CleanupDUnitVMsRule bounces VMs before and after each test. >>=20 >> Problematically, JUnit rules applied in an unpredictable / = JVM-dependent >> ordering. [2] As a result, some flakiness we find in our tests may = be the >> result of unexpected interaction and ordering of our test rules. [3] >>=20 >> Fortunately, a solution to this problem already exists. Rule = ordering can >> be imposed by JUnit's RuleChain. [4] >>=20 >> In early exploration with this rule, some tests failed due to the = RuleChain >> not being serializable. However, as it should only apply to the test = VM, >> and given that it can be composed of (unannotated) rules that remain >> accessible and serializable, it should be a simple matter of = declaring the >> offending field transient, as it will only be necessary in the test = VM. >>=20 >> So, you dear reader: while you're out there making Geode the best it = can >> be, if you find yourself in a test class that uses more than one rule >> listed in [1], or if you notice some other rule not listed below that >> reaches out to VMs as part of its @Before or @After, please update = that >> test to use the RuleChain to apply the rules in a predictable order. >>=20 >> Imagination is Change. >> ~Patrick >>=20 >> [1] A probably-incomplete list of invasive rules can be found via >> $> git grep -il inEveryVM | grep Rule.java >>=20 >> = geode-core/src/distributedTest/java/org/apache/geode/management/Management= TestRule.java >> = geode-dunit/src/main/java/org/apache/geode/test/dunit/rules/CacheRule.java= >>=20 >> = geode-dunit/src/main/java/org/apache/geode/test/dunit/rules/ClientCacheRul= e.java >>=20 >> = geode-dunit/src/main/java/org/apache/geode/test/dunit/rules/DistributedDis= kDirRule.java >>=20 >> = geode-dunit/src/main/java/org/apache/geode/test/dunit/rules/DistributedRul= e.java >>=20 >> = geode-dunit/src/main/java/org/apache/geode/test/dunit/rules/DistributedUse= JacksonForJsonPathRule.java >>=20 >> = geode-dunit/src/main/java/org/apache/geode/test/dunit/rules/SharedCounters= Rule.java >>=20 >> [2] See the documentation for rules here: >> https://junit.org/junit4/javadoc/4.12/org/junit/Rule.html ; notably, >> "However, >> if there are multiple [Rule] fields (or methods) they will be applied = in an >> order that depends on your JVM's implementation of the reflection = API, >> which is undefined, in general." >>=20 >> [3] For what it's worth, this was discovered after looking into why = the >> DistributedRule check fo suspicious strings caused the test *after* = the >> test that emitted the strings to fail. It's only tangentially = related, but >> got me looking into when and how the @After was getting applied. >>=20 >> [4] = https://junit.org/junit4/javadoc/4.12/org/junit/rules/RuleChain.html >>=20 --Apple-Mail=_C13B5581-49DA-47F7-A327-756F9950B2D5--