ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ilya Kasnacheev <ilya.kasnach...@gmail.com>
Subject Re: Flaky tests
Date Wed, 27 Feb 2019 09:01:00 GMT

We also have this wonderful tool: https://mtcga.gridgain.com/

It keeps track of tests which are currently flaky or under investigation,
and given your change it will report non-flaky tests that failed, skipping
all problematic tests.

It talks to TeamCity so it can run tests for you, check their results and
comment JIRA with "Bot Visa", see:
https://issues.apache.org/jira/browse/IGNITE-11299 comment from "Ignite TC

Since we don't have too many flaky tests thanks to Make TeamCity Green
Again movement, it works reasonably well.

A year ago we did not have any of this but now we do, and we were able to
shave 80% of test-checking busy work. This is mostly thanks to Dmitry

Ilya Kasnacheev

ср, 27 февр. 2019 г. в 01:39, Stanislav Kozlovski <

> Hey there Ignite community,
> I contribute to a fellow open-source project - Apache Kafka - and there we
> have been fighting flaky tests a lot. We run Java 8 and Java 11 builds on
> every Pull Request and due to test flakiness, almost all of them turn out
> red with 1 or 2 tests (completely unrelated to the change in the PR)
> failing. This has resulted in committers either ignoring them and merging
> the changes or in the worst case rerunning the hour-long build until it
> becomes green.
> This test flakiness has also slowed down our releases significantly.
> In general, I was just curious to understand if this is a problem that
> your project faces as well. Does your project have a lot of intermittently
> failing tests, do you have any active process of addressing such tests
> (during the initial review, after realizing it is flaky, etc). Any pointers
> will be greatly appreciated!
> Thanks,
> Stanislav
> <http://aka.ms/weboutlook>

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message