From dev-return-312410-archive-asf-public=cust-asf.ponee.io@lucene.apache.org Thu Feb 22 00:08:59 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id F371018061A for ; Thu, 22 Feb 2018 00:08:58 +0100 (CET) Received: (qmail 94432 invoked by uid 500); 21 Feb 2018 23:08:52 -0000 Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@lucene.apache.org Delivered-To: mailing list dev@lucene.apache.org Received: (qmail 94411 invoked by uid 99); 21 Feb 2018 23:08:51 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Feb 2018 23:08:51 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 7D3BD1A0855 for ; Wed, 21 Feb 2018 23:08:51 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.679 X-Spam-Level: X-Spam-Status: No, score=0.679 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id oXEX8ojdd24p for ; Wed, 21 Feb 2018 23:08:50 +0000 (UTC) Received: from mail-wm0-f53.google.com (mail-wm0-f53.google.com [74.125.82.53]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id D04A95F47E for ; Wed, 21 Feb 2018 23:08:49 +0000 (UTC) Received: by mail-wm0-f53.google.com with SMTP id z81so449575wmb.4 for ; Wed, 21 Feb 2018 15:08:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=aiiIJ6fiFo97guttiBZ8uoKXCOU+BXGugYFS0wdwVlo=; b=uBCGuCx15nH+sAGw/RlUD7VCjzYxEnfCqvTRHDCwVA9EKYh0gNKwWfh0IwlMX9RXa/ vFxWqjEPV6PVlbUT6Y4yfhGNoHF1HDRCGujqfFlOymux7S7V0QfP6qNMeYGi9AxsfuRh ILx156q3iOsEJzJipCAc1oJHZ2HPuYuvAGx32NoDDy/KKo0g2z0FkGMSXwp1D8pOLkAi xnmHYLMBW3Z2tYBvStaBpSop8vdeXqsH5o5Fkp6h0dj7xVAXheJt92rFr7bLi/CGsDEc 0OHYGXa5Ht0XY5+qdsSeiDJpIG0RqQwItSGxP09BDi4jD0QUnbwfEQGIsug/Af2cOWzr P8SA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=aiiIJ6fiFo97guttiBZ8uoKXCOU+BXGugYFS0wdwVlo=; b=GiXp2P7rkLfgAhGRUHUufcX9Fr11cq7klKurbbsh7yrzqtCDObBMonAG6ZI6PMROcw lehWSiGedURt1pRJ1zJk68AdhdN7Jq2XdzazAUmFSdAIPrOfF4SBxBnq10OUObk047iG /G07JwmdATWA0lvvRPB7vNBLXrm2Pb/ZN5sgGA9mv0/8BXQ12ILtIHlV27Nu10z3zLWQ Y6uPgPcCfJzLfIbPmWR78uqK+MaMfGQpxHKY9X0mqIPtwXNspee3rXNL+47tP+sVpyZ3 uGpf+lMO667B12uTb1AY9YDqFnZZudkrcnqXQvcrre4wuZwcuXkfTFLUfCzNYnr8HqMs 7HTQ== X-Gm-Message-State: APf1xPAkIRPolTzejYMcunEtA9oLPXE59rqyVyUnKGMpyVg3lRIX2EjF goXeeL0X3SNLdML++oIIPcv3QHSQFcuZ7T5+ghJEIQ== X-Google-Smtp-Source: AH8x226BE+AnMdRusO80XFndKb9fgbgebrpdpE7BmHm7zb573WCwPHTTEdAuZpcR25YGOLaNNMVPGWuPGK+VMTP11Ic= X-Received: by 10.28.163.5 with SMTP id m5mr3144272wme.7.1519254529093; Wed, 21 Feb 2018 15:08:49 -0800 (PST) MIME-Version: 1.0 Received: by 10.28.196.73 with HTTP; Wed, 21 Feb 2018 15:08:48 -0800 (PST) In-Reply-To: <012701d3ab68$2c3c3c90$84b4b5b0$@thetaphi.de> References: <012701d3ab68$2c3c3c90$84b4b5b0$@thetaphi.de> From: Jason Gerlowski Date: Wed, 21 Feb 2018 18:08:48 -0500 Message-ID: Subject: Re: Test failures are out of control...... To: dev@lucene.apache.org Content-Type: text/plain; charset="UTF-8" I don't have strong opinions about what we do with our existing flaky tests. I think re-running failures before commit might theoretically catch more bugs than ignoring the test outright, but with all the noise and how standard it is to need to rerun tests I'd be surprised if the numbers are all that different. Where I see some potential common ground though is in preventing new flaky tests. And in the long run, I think what we do to prevent new flakes is going to be much more important than how we handle the BadApples we have at this particular instant. If we can put our finger in the dam, the existing flakiness becomes much easier to put a dent in. I'm curious what you guys had in mind when you mentioned preventing new flaky tests from popping up. What are our options for "enforcing" that? Were you imagining reopening JIRAs and asking the original committer to investigate? Or outright reverting commits that introduce flaky tests? Or something in between (like disabling features with flaky tests prior to releases)? Best, Jason On Wed, Feb 21, 2018 at 6:03 PM, Uwe Schindler wrote: > Hi, > >> Flakey Test Problems: >> a) Flakey tests create so much noise that people no longer pay >> attention to the automated reporting via email. >> b) When running unit tests manually before a commit (i.e. "ant test") >> a flakey test can fail. >> >> Solutions: >> We cloud fix (a) by marking as flakey and having a new target >> "non-flakey" that is run by the jenkins jobs that are currently run >> continuously. > > We have a solution for this already: Mark all those tests with @AwaitsFix or @BadApple > By default those aren't executed in Jenkins runs and also not for developers, but devs can enable/disable them using -Dtests.awaitsfix=true and -Dtests.badapples=true: > > [help] # Test groups. ---------------------------------------------------- > [help] # > [help] # test groups can be enabled or disabled (true/false). Default > [help] # value provided below in [brackets]. > [help] > [help] ant -Dtests.nightly=[false] - nightly test group (@Nightly) > [help] ant -Dtests.weekly=[false] - weekly tests (@Weekly) > [help] ant -Dtests.awaitsfix=[false] - known issue (@AwaitsFix) > [help] ant -Dtests.slow=[true] - slow tests (@Slow) > > We can of course also make a weekly jenkins jobs that enables those tests on Jenkins only weekly (like nightly stuff). We have "tests.badapples" and "tests.awaitsfix" - I don't know what's the difference between both. > > So we have 2 options to classify tests, let's choose one and apply it to all Flakey tests! > > Uwe > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org > For additional commands, e-mail: dev-help@lucene.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org