mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marco de Abreu <marco.g.ab...@googlemail.com>
Subject Re: PR validation and runtime of CI
Date Thu, 07 Jun 2018 15:27:52 GMT
Yeah, I think we are at the point at which we have to disable tests..

If a test fails in nightly, the commit would not be reverted since it's
hard to pin a failure to a specific PR. We will have reporting for failures
on nightly (they have proven to be stable, so we can enable it right from
the beginning). I'm also currently working on a system that tracks all test
failures, so this will also cover nightly tests. This will give us
actionable data which allows us to define acceptance criteria for a
release. E.g. if the test success rate is below X%, a release can not be
made. This could be set to 100% for nightly, for example.

It would definitely be good if we could determine which tests are required
to run and which ones are unnecessary. I don't really like the flag in the
comment (and also it's hard to integrate). A good idea would be some
analytics on the changed file content. If we have this data, we could
easily enable and disable different jobs. Since this behaviour is entirely
defined in GitHub, I'd like to invite everybody to submit a PR.

-Marco



On Thu, Jun 7, 2018 at 5:20 PM Aaron Markham <aaron.s.markham@gmail.com>
wrote:

> I'd like to disable flaky tests until they're fixed.
> What would the process be for fixing a failure if the tests are done
> nightly? Would the commit be reverted? Won't we end up in the same
> situation with so many flaky tests?
>
> I'd like to see if we can separate the test pipelines based on the content
> of the commit. I think that md, html, and js updates should fly through and
> not have to go through GPU tests.
>
> Maybe some special flag added to the comment?
> Is this possible?
>
>
> On Wed, Jun 6, 2018 at 10:37 PM, Pedro Larroy <
> pedro.larroy.lists@gmail.com>
> wrote:
>
> > Hi Team
> >
> > The time to validate a PR is growing, due to our number of supported
> > platforms and increased time spent in testing and running models.  We are
> > at approximately 3h for a full successful run.
> >
> > This is compounded with the failure rate of builds due to flaky tests of
> > more than 50% which is a big drag in developer productivity if you can
> only
> > get one or two CI runs to a change per day.
> >
> > I would want a turnaround time of less than 30 minutes and 0% failure
> rate
> > on master.
> >
> > For this I propose working towards moving tests from CI to nightly,
> > specially the ones that take most time or do black box testing with full
> > training of models. And addressing flaky tests by either fixing them or
> > disabling them.
> >
> > I would like to check if there's consensus on this previous plan so we
> are
> > aligned on pursuing this common goal as a shared effort.
> >
> > Pedro.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message