mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marco de Abreu <>
Subject Re: [CI] Staggered build pipelines enabled
Date Mon, 15 Jun 2020 23:19:51 GMT

I'd like to revisit this decision and review whether the expected benefit
(cost reduction) was achieved and how the overall PR validation duration
has changed. Could you guys share some information on this matter?

Just today, we've had two incidents which were caused by this change:

1. This PR was merged prematurely because the follow up pipelines didn't
run (whether it's a timing issue or something else I don't know). We've
always had the policy of not enforcing protected feature branches and this
pipeline is causing human errors.

2. The sanity check, which got into the critical path here, starts running
into timeouts. I'm aware that this is in case of cache misses, but none the
less does that heavily increase the waiting duration since developers now
have to wait for two entire cache build cycles instead of just one:

I understand that money has to be conserved, but I still stand by my
opinion that this was the wrong move and development speed was sacrificed.

If there are no other compelling arguments, I'd prefer if the previous
state of parallel pipelines could be restored.

Best regards

sandeep krishnamurthy <> schrieb am Di., 28.
Apr. 2020, 07:55:

> Thanks a lot Joe for your contributions. Thank you Marco, Chai and Leo for
> helping this.
> Especially given that you had seen around 57% build failing in sanity
> check, this should be very helpful to provide faster feedback for PR
> authors on sanity issues plus save a lot of unnecessary builds.
> Best,
> Sandeep
> On Mon, 27 Apr 2020, 10:20 pm Joe Evans, <> wrote:
> > Hi dev community,
> >
> >
> > We have made the changes to the mxnet CI system to incorporate the
> > staggered build pipelines. With this change, when a new PR is created or
> an
> > existing PR is updated, the status checks will only show
> > “ci/jenkins/mxnet-validation/sanity” build job at first. Once this build
> > completes successfully (avg. run time is about 10min), the remaining CI
> > build jobs will appear and function as previously.
> >
> >
> > Please let me know if you experience any issues with this change.
> >
> >
> > Thanks!
> >
> > Joe
> >
> >
> > References:
> >
> >
> >
> >

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message