reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Julia Wang (QIUHE)" <Qiuhe.W...@microsoft.com.INVALID>
Subject RE: Disable tests with transient failures
Date Tue, 18 Jul 2017 23:28:12 GMT
I would keep e2e tests, those are important tests that ensure the system is still function.
It will detects issues if we  change the bridge code or contract. It helps us to find any
regression that may be caused from Java side change and doesn't work from .Net. Most unit
tests use mock, they only tests the code path without complete e2e scenario. In the past years,
we added so many e2e functional tests, mainly used to detect regression issues and ensure
we are not badly broken. 

Transit test issues are separate. Depending on log files to verify the result is one issue,
but potentially we have some driver not ends properly issue I think. Recently I got a few
repro in my local test run, when it happened, apparently, the system was still running and
never ended until I stopped it. We should not ignore those issues. However, I also don't think
they must be the blocker of releases. We should allow bugs in a release, as long as the bugs
don't impact the major functionality of the system and we track the bugs. 

Julia

-----Original Message-----
From: Douglas Service [mailto:dsopsrc@gmail.com] 
Sent: Tuesday, July 18, 2017 3:18 PM
To: dev@reef.apache.org
Subject: Re: Disable tests with transient failures

I would only disable the ones with transient failures as we need some level of testing.

On Tue, Jul 18, 2017 at 2:31 PM, Markus Weimer <markus@weimo.de> wrote:

> Coming back to the original question: I am actually +1 on disabling
> *all* integration tests in REEF.NET.
>
> Reasoning: The integration tests are flaky, and we don't really know 
> why. It is plausible that our test approach based on log files is to 
> blame for the failures. We don't even trust them and always run 
> HelloREEF or such on a YARN cluster to know whether or not we broke 
> something. At the same time, we know that our current approach to 
> integration tests in REEF.NET is broken and at best allows us to make 
> statements about the local runtime. Meanwhile, the failing tests 
> prevent releases which in turn prevent us from moving on the work 
> towards REEF.NET on Linux, which would broaden the developer base and 
> therefore help with testing. This creates a situation where our bad 
> approach to testing prevents us from making progress towards quality 
> software, which is the opposite of what tests should do.
>
> Hence, I propose the following:
>
>   1. Disable all REEF.NET integration tests. Leave them in the code 
> base for reference, but ignore them when doing CI builds or releases.
>   2. Make a release with zero integration tests of REEF.NET. Be open 
> about the fact that we have no integration tests that convince us or 
> anyone that REEF.NET actually works.
>   3. Add integration tests to REEF.NET, one at a time. This time, we 
> make sure that they are solid and indicative of actual quality of the 
> code.
>
>
> WDYT?
>
> Markus
>
Mime
View raw message