mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anton Chernov <mecher...@gmail.com>
Subject Re: Splitting Jenkins pipelines - stop changes to Jenkinsfiles!
Date Wed, 21 Nov 2018 16:42:34 GMT
The ability to retrigger the pipelines separately is an amazing step
forward. Great job Marco!

ср, 21 нояб. 2018 г. в 15:03, Marco de Abreu
<marco.g.abreu@googlemail.com.invalid>:

> Hello,
>
> the PR has been merged and I've created the new pipelines at [1]. You can
> see the new reports if you have a look at this example PR at [2].
>
> The new status messages will be the ones starting with
> "ci/jenkins/mxnet-validation/".
>
> This now allows you to retrigger specific pipelines if they fail. For
> example, if you're interested in the website pipeline, you can now go to
> [3] and just retrigger that instead of running the entire suite. Whenever
> there's a new commit, all pipelines will still be scheduled as before (the
> overall behaviour or coverage of our pipeline did not change, I just
> decoupled them and increased the usability).
>
> The next step will be the deprecation of the main Jenkinsfile (the one
> which reports the status as "continuous-integration/jenkins/pr-merge") and
> requesting these new statuses to be marked as required (protected master
> branch). Since we have to change some reporting tools to point to the new
> jobs and I'd like to observe the stability for some time, this will take
> some times.
>
> You can now resume changes in the Jenkinsfiles. But please do not modify
> the Jenkinsfile in the root directory but instead the ones at [4]. The
> nightly Jenkinsfiles (or basically all Jenkinsfiles that are not part of
> the main pipeline) have not been migrated yet and I will do that at a later
> point in time.
>
> Best regards,
> Marco
>
> [1]: http://jenkins.mxnet-ci.amazon-ml.com/job/mxnet-validation/
> [2]: https://github.com/apache/incubator-mxnet/pull/13352
> [3]:
>
> http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fwebsite/detail/PR-13352/1/pipeline
> [4]: https://github.com/apache/incubator-mxnet/tree/master/ci/jenkins
>
> On Tue, Nov 20, 2018 at 9:33 PM Marco de Abreu <
> marco.g.abreu@googlemail.com>
> wrote:
>
> > I have just submitted my PR at
> > https://github.com/apache/incubator-mxnet/pull/13344. Test jobs are
> > available at
> > http://jenkins.mxnet-ci-dev.amazon-ml.com/view/test-marco-mxnet/.
> >
> > As soon as I'm done with my tests, I will mark it as ready for review.
> >
> > Best regards,
> > Marco
> >
> > On Tue, Nov 20, 2018 at 9:09 PM Marco de Abreu <
> > marco.g.abreu@googlemail.com> wrote:
> >
> >> Thanks, Pedro!
> >>
> >> I have also been looking into that issue, but it seems like this would
> >> require changes in the groovy interpreter of Jenkins. From what I can
> tell,
> >> a refactor will give us multiple benefits (clarity and speed) aside from
> >> resolving this issue.
> >>
> >> Best regards,
> >> Marco
> >>
> >> Am Di., 20. Nov. 2018, 19:54 hat Pedro Larroy <
> >> pedro.larroy.lists@gmail.com> geschrieben:
> >>
> >>> I think this is a big problem, which has blocked us before. I want to
> >>> point out that you are doing a great thing by avoiding everyone
> >>> getting blocked by refactoring the pipelines.
> >>>
> >>> My concern is that we are kicking the can down the road and not
> >>> addressing the root cause of the problem with is known
> >>> https://issues.jenkins-ci.org/browse/JENKINS-37984
> >>>
> >>> Pedro.
> >>>
> >>>
> >>> On Tue, Nov 20, 2018 at 6:08 PM Marco de Abreu
> >>> <marco.g.abreu@googlemail.com.invalid> wrote:
> >>> >
> >>> > Hello Steffen,
> >>> >
> >>> > no, there won't be any impact on the PR process or nightly
> regressions.
> >>> > Only the reporting will have to be updated with the new job links,
> but
> >>> that
> >>> > should be a minor issue. To avoid any outage, I have been thinking
> >>> about
> >>> > running both versions in parallel.
> >>> >
> >>> > Best regards,
> >>> > Marco
> >>> >
> >>> >
> >>> >
> >>> > On Tue, Nov 20, 2018 at 5:53 PM Steffen Rochel <
> >>> steffenrochel@gmail.com>
> >>> > wrote:
> >>> >
> >>> > > Hi Marco - is there any impact on reporting, the PR process or
> >>> nightly
> >>> > > regression beside reduction in TAT?  If yes, please elaborate.
> >>> > > Steffen
> >>> > >
> >>> > > On Tue, Nov 20, 2018 at 8:05 AM Marco de Abreu
> >>> > > <marco.g.abreu@googlemail.com.invalid> wrote:
> >>> > >
> >>> > > > Hello,
> >>> > > >
> >>> > > > we ran into issues around the maximum filesize of the Jenkinsfile
> >>> a few
> >>> > > > times already. In order to resolve this issue, I'd like to
> combine
> >>> this
> >>> > > > with some refactors I have planned for quite some time.
> >>> > > >
> >>> > > > The idea is basically to move away from one big Jenkinsfile
and
> >>> instead
> >>> > > > split it into separate jobs that run in parallel and report
their
> >>> status
> >>> > > > individually. Besides avoiding the size restriction, this
will
> >>> greatly
> >>> > > > speed up the PR validation process by reducing the critical
path.
> >>> Instead
> >>> > > > of having to wait for every single step within a stage to
finish
> >>> before
> >>> > > the
> >>> > > > next stage (e.g. tests) is getting executed, these pipelines
> would
> >>> now be
> >>> > > > able to move forward individually. I'm still in the process
of
> >>> > > refactoring
> >>> > > > and can't provide any numbers or documentation at this time,
but
> I
> >>> would
> >>> > > > like to announce this early on to avoid conflicts:
> >>> > > >
> >>> > > > Since I will remove the original Jenkinsfile, this might
cause
> >>> conflicts
> >>> > > > with ongoing efforts that try to change the Jenkinsfile.
This
> >>> poses the
> >>> > > > risk that I might forget to port a change. Thus, I'd like
to ask
> >>> all
> >>> > > > contributors to wait with changes of Jenkinsfile and would
like
> to
> >>> > > request
> >>> > > > fellow-committers to wait with merging any Jenkinsfile-related
> PRs
> >>> until
> >>> > > > further notice.
> >>> > > >
> >>> > > > I expect to finish this refactor until the end of the week.
> Please
> >>> don't
> >>> > > > hesitate to ask if you've got further questions.
> >>> > > >
> >>> > > > Please excuse any caused inconveniences.
> >>> > > >
> >>> > > > Best regards,
> >>> > > > Marco
> >>> > > >
> >>> > >
> >>>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message