reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Douglas Service <dsop...@gmail.com>
Subject Re: Disable tests with transient failures
Date Tue, 18 Jul 2017 22:17:31 GMT
I would only disable the ones with transient failures as we need some level
of testing.

On Tue, Jul 18, 2017 at 2:31 PM, Markus Weimer <markus@weimo.de> wrote:

> Coming back to the original question: I am actually +1 on disabling
> *all* integration tests in REEF.NET.
>
> Reasoning: The integration tests are flaky, and we don't really know
> why. It is plausible that our test approach based on log files is to
> blame for the failures. We don't even trust them and always run
> HelloREEF or such on a YARN cluster to know whether or not we broke
> something. At the same time, we know that our current approach to
> integration tests in REEF.NET is broken and at best allows us to make
> statements about the local runtime. Meanwhile, the failing tests
> prevent releases which in turn prevent us from moving on the work
> towards REEF.NET on Linux, which would broaden the developer base and
> therefore help with testing. This creates a situation where our bad
> approach to testing prevents us from making progress towards quality
> software, which is the opposite of what tests should do.
>
> Hence, I propose the following:
>
>   1. Disable all REEF.NET integration tests. Leave them in the code
> base for reference, but ignore them when doing CI builds or releases.
>   2. Make a release with zero integration tests of REEF.NET. Be open
> about the fact that we have no integration tests that convince us or
> anyone that REEF.NET actually works.
>   3. Add integration tests to REEF.NET, one at a time. This time, we
> make sure that they are solid and indicative of actual quality of the
> code.
>
>
> WDYT?
>
> Markus
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message