From dev-return-312409-archive-asf-public=cust-asf.ponee.io@lucene.apache.org Thu Feb 22 00:03:28 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 45A7D18061A for ; Thu, 22 Feb 2018 00:03:28 +0100 (CET) Received: (qmail 86235 invoked by uid 500); 21 Feb 2018 23:03:26 -0000 Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@lucene.apache.org Delivered-To: mailing list dev@lucene.apache.org Received: (qmail 86218 invoked by uid 99); 21 Feb 2018 23:03:26 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Feb 2018 23:03:26 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id CF1991805BD for ; Wed, 21 Feb 2018 23:03:25 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.002 X-Spam-Level: X-Spam-Status: No, score=-0.002 tagged_above=-999 required=6.31 tests=[RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 4LroRnD_YOqN for ; Wed, 21 Feb 2018 23:03:24 +0000 (UTC) Received: from mail.sd-datasolutions.de (serv2.sd-datasolutions.de [78.47.65.36]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 516415F1B3 for ; Wed, 21 Feb 2018 23:03:24 +0000 (UTC) Received: from VEGA (p200300C103E7910178DEF99FA9557441.dip0.t-ipconnect.de [IPv6:2003:c1:3e7:9101:78de:f99f:a955:7441]) by mail.sd-datasolutions.de (Postfix) with ESMTPSA id C3B31480FA0 for ; Wed, 21 Feb 2018 23:03:22 +0000 (UTC) X-NSA-Greeting: Dear NSA, have fun with reading and analyzing this e-mail! From: "Uwe Schindler" To: References: In-Reply-To: Subject: RE: Test failures are out of control...... Date: Thu, 22 Feb 2018 00:03:22 +0100 Message-ID: <012701d3ab68$2c3c3c90$84b4b5b0$@thetaphi.de> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Mailer: Microsoft Outlook 16.0 Thread-Index: AQIpf+JmOyN3AvCI9SwPc57xzQeRBwK/C5v9AhKc7lYB4GHWLwJjOM2sAIpd0sICuvHY5aKg+cdw Content-Language: de Hi, > Flakey Test Problems: > a) Flakey tests create so much noise that people no longer pay > attention to the automated reporting via email. > b) When running unit tests manually before a commit (i.e. "ant test") > a flakey test can fail. >=20 > Solutions: > We cloud fix (a) by marking as flakey and having a new target > "non-flakey" that is run by the jenkins jobs that are currently run > continuously. We have a solution for this already: Mark all those tests with = @AwaitsFix or @BadApple By default those aren't executed in Jenkins runs and also not for = developers, but devs can enable/disable them using = -Dtests.awaitsfix=3Dtrue and -Dtests.badapples=3Dtrue: [help] # Test groups. = ---------------------------------------------------- [help] # [help] # test groups can be enabled or disabled (true/false). = Default [help] # value provided below in [brackets]. [help] [help] ant -Dtests.nightly=3D[false] - nightly test group = (@Nightly) [help] ant -Dtests.weekly=3D[false] - weekly tests (@Weekly) [help] ant -Dtests.awaitsfix=3D[false] - known issue (@AwaitsFix) [help] ant -Dtests.slow=3D[true] - slow tests (@Slow) We can of course also make a weekly jenkins jobs that enables those = tests on Jenkins only weekly (like nightly stuff). We have = "tests.badapples" and "tests.awaitsfix" - I don't know what's the = difference between both. So we have 2 options to classify tests, let's choose one and apply it to = all Flakey tests! Uwe --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org