From dev-return-44990-archive-asf-public=cust-asf.ponee.io@ignite.apache.org Wed Feb 27 12:13:29 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 61892180608 for ; Wed, 27 Feb 2019 13:13:29 +0100 (CET) Received: (qmail 50124 invoked by uid 500); 27 Feb 2019 12:13:28 -0000 Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ignite.apache.org Delivered-To: mailing list dev@ignite.apache.org Received: (qmail 50113 invoked by uid 99); 27 Feb 2019 12:13:28 -0000 Received: from mail-relay.apache.org (HELO mailrelay1-lw-us.apache.org) (207.244.88.152) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Feb 2019 12:13:28 +0000 Received: from mail-qk1-f173.google.com (mail-qk1-f173.google.com [209.85.222.173]) by mailrelay1-lw-us.apache.org (ASF Mail Server at mailrelay1-lw-us.apache.org) with ESMTPSA id 7C22A128A for ; Wed, 27 Feb 2019 12:13:27 +0000 (UTC) Received: by mail-qk1-f173.google.com with SMTP id x9so9689181qkf.0 for ; Wed, 27 Feb 2019 04:13:27 -0800 (PST) X-Gm-Message-State: AHQUAuY0MlKDXQ0tq8OnEhq7/Pd2EG1kfgcH73IOvD/zsRjMKgWgM2cW jIK/s4umlGqZXtqWU0wjBoGjflWfpUar8qO83/o= X-Google-Smtp-Source: AHgI3IZA35kAnLByQenjLAxglubGo67VbVUwaN7erqx9GDSI3MVX13TkFMdqCv2hNME7bwQXCOIt3RAlVeC84kpxCyw= X-Received: by 2002:a37:a09:: with SMTP id 9mr1767206qkk.341.1551269607185; Wed, 27 Feb 2019 04:13:27 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Dmitriy Pavlov Date: Wed, 27 Feb 2019 15:13:17 +0300 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Flaky tests To: dev@ignite.apache.org Content-Type: multipart/alternative; boundary="000000000000e3248c0582df1968" --000000000000e3248c0582df1968 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thanks, Ilya for your positive feedback. Stanislav, Apache Ignite TeamCity Bot requires CI credentials, so it may be easier to check its code and description here https://github.com/apache/ignite-teamcity-bot#ignite-teamcity-bot (For Russian native speakers there is a blog post here https://habr.com/ru/company/sberbank/blog/436070/ , it is not yet translated to English) As for Java 11, there is an idea to separate stable/flaky statistics from the master for two Java versions, but it is only planned now. Sincerely, Dmitriy Pavlov =D1=81=D1=80, 27 =D1=84=D0=B5=D0=B2=D1=80. 2019 =D0=B3. =D0=B2 12:01, Ilya = Kasnacheev : > Hello! > > We also have this wonderful tool: https://mtcga.gridgain.com/ > > It keeps track of tests which are currently flaky or under investigation, > and given your change it will report non-flaky tests that failed, skippin= g > all problematic tests. > > It talks to TeamCity so it can run tests for you, check their results and > comment JIRA with "Bot Visa", see: > https://issues.apache.org/jira/browse/IGNITE-11299 comment from "Ignite T= C > Bot" > > Since we don't have too many flaky tests thanks to Make TeamCity Green > Again movement, it works reasonably well. > > A year ago we did not have any of this but now we do, and we were able to > shave 80% of test-checking busy work. This is mostly thanks to Dmitry > Pavlov. > > Regards, > -- > Ilya Kasnacheev > > > =D1=81=D1=80, 27 =D1=84=D0=B5=D0=B2=D1=80. 2019 =D0=B3. =D0=B2 01:39, Sta= nislav Kozlovski < > Stanislav_Kozlovski@outlook.com>: > > > Hey there Ignite community, > > > > I contribute to a fellow open-source project - Apache Kafka - and there > we > > have been fighting flaky tests a lot. We run Java 8 and Java 11 builds = on > > every Pull Request and due to test flakiness, almost all of them turn o= ut > > red with 1 or 2 tests (completely unrelated to the change in the PR) > > failing. This has resulted in committers either ignoring them and mergi= ng > > the changes or in the worst case rerunning the hour-long build until it > > becomes green. > > This test flakiness has also slowed down our releases significantly. > > > > In general, I was just curious to understand if this is a problem that > > your project faces as well. Does your project have a lot of > intermittently > > failing tests, do you have any active process of addressing such tests > > (during the initial review, after realizing it is flaky, etc). Any > pointers > > will be greatly appreciated! > > > > Thanks, > > Stanislav > > > > > > > > > --000000000000e3248c0582df1968--