mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kellen sunderland <>
Subject Re: CI impaired
Date Mon, 26 Nov 2018 01:47:34 GMT
Sorry, [1] meant to reference .

On Sun, Nov 25, 2018 at 5:41 PM kellen sunderland <> wrote:

> Marco and I ran into another urgent issue over the weekend that was
> causing builds to fail.  This issue was unrelated to any feature
> development work, or other CI fixes applied recently, but it did require
> quite a bit of work from Marco (and a little from me) to fix.
> We spent enough time on the problem that it caused us to take a step back
> and consider how we could both fix issues in CI and support the 1.4 release
> with the least impact possible on MXNet devs.  Marco had planned to make a
> significant change to the CI to fix a long-standing Jenkins error [1], but
> we feel that most developers would prioritize having a stable build
> environment for the next few weeks over having this fix in place.
> To properly introduce a new CI system the intent was to do a gradual
> blue/green roll out of the fix.  To manage this rollout would have taken
> operational effort and double compute load as we run systems in parallel.
> This risks outages due to scaling limits, and we’d rather make this change
> during a period of low-developer activity, i.e. shortly after the 1.4
> release.
> This means that from now until the 1.4 release, in order to reduce
> complexity MXNet developers should only see a single Jenkins verification
> check, and a single Travis check.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message