mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Olivier <>
Subject Re: Random data in tests produces unstable test coverage
Date Sun, 20 Jan 2019 16:54:50 GMT
wouldn’t abandoning stochastic testing mean that execution paths would no
longer be tested and therefore increase issues found in production?

On Sun, Jan 20, 2019 at 5:13 AM Marco de Abreu <>

> Hello everyone,
> a few months ago, I have enabled test coverage recording for the MXNet
> repository [1]. The next step would be to report the coverage changes to
> pull requests so we can make them part of the review. For this, I've
> created a pull request here at [2].
> Before this feature can be enabled though, we have to stabilize the test
> coverage. Due to the extensive usage of random inputs in our tests, the
> execution takes different paths and thus, the coverage results undergo a
> heavy variance. If we would enable the coverage report for PRs, they would
> be marked as increased/decreased coverage without actually doing so. To get
> a few examples, visit [3].
> I'd like to provide some detailed examples or screenshots, but with big
> reports [4], CodeCov's webinterface tends to time out. I currently have an
> open ticket with CodeCov that will hopefully improve the situation soon,
> but we have to live with the timeouts until then.
> My proposal to improve the situation would be to divide the work across the
> community in two stages:
> 1. Replace random inputs in tests with deterministic ones if applicable.
> 2. If randomness cannot be prevented or this 'flakiness' persists, we could
> run the same test multiple times with coverage, look at the coverage diff
> and then write targeted unittests for the inconsistent parts.
> If there is more interest towards test coverage, I'd be happy to write a
> guide that explains how to measure test coverage and detect flaky coverage
> areas so everybody can help contribute towards a stable test coverage.
> Please let me know what you think.
> Best regards,
> Marco
> [1]:
> [2]:
> [3]:
> [4]:

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message