geode-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Murmann <>
Subject Re: [DISCUSS] Maximum duration that a class may contain a @Flaky
Date Mon, 06 Nov 2017 17:36:31 GMT
Everything that's in flaky should be tracked in some way. When we add
something to flaky tests, someone should at least passively be working on
it, e.g. gathering more information by adding logs or similar.
For everything that's already in flaky we should make an effort to work
through them and make them not flaky or get test coverage for the
implementation code via different tests. So eventually there should be
nothing in flaky that is not on somebodies radar. We might get some kind of
a tragedy of the commons issue with tests that nobody feels responsible
for. Having more tests fail likely won't solve that though, since it still
doesn't assign responsibility.

I think this time is better spent tackling flaky tests that are related to
the code you feel responsible for. Once we have cut down significantly on
our amount of flaky tests we should sit down and split up the orphaned
flaky tests. In the meantime the current 400+ flaky tests provide plenty
opportunity to improve the situation.

On Fri, Nov 3, 2017 at 10:33 AM, Patrick Rhomberg <>

> Hello, all!
>   I was considering doing some git archeology centered around identifying
> how long a any given test class containing a @Flaky has had that
> annotation.  Ultimately, I think it would be good to add a test that would
> fail when any one test has been flaky for too long.  I feel like many of
> our flaky tests have fallen by the wayside, and this could provide the
> impetus to resolve these issues in a timely fashion.
>   This leads naturally to the question: How long should a test be allowed
> to remain marked Flaky?  Certainly, flaky tests are most often of the
> non-deterministic, hard-to-reproduce variety, so some leeway is deserved.
> Two weeks?  One month?
>   Thoughts?
> Imagination is Change.
> ~Patrick Rhomberg

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message