mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marco de Abreu <marco.g.ab...@googlemail.com.INVALID>
Subject Re: Splitting Jenkins pipelines - stop changes to Jenkinsfiles!
Date Wed, 21 Nov 2018 23:49:24 GMT
Hi Anirudh,

I have just deployed a fix for this. The status reports are now back to
where they were:

- "continuous-integration/jenkins/pr-merge" is the pipeline everybody is
used and backed by the main Jenkinsfile -> This is the one you have to look
at
- "ci/jenkins/mxnet-validation/XXX" this is the new pipeline. It's 100%
equivalent with the existing pipeline. It's completely working, but you can
ignore it for now.
- "continuous-integration/travis-ci/pr" this is the travis pipeline which
is currently in beta, you can ignore it for now

This means that the status report with the "required" flag is really
required and you can ignore everything else.

I will respond the CI status question in a separate thread.

Best regards,
Marco


On Thu, Nov 22, 2018 at 12:25 AM Anirudh <anirudh2290@gmail.com> wrote:

> Hi Marco,
>
> Can you point out specifically which checks we have to make sure pass
> before merging PRs. Currently apart from the required one there are six
> steps added.  Also, is the CI down currently :
>
> http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-13324/17/pipeline
>
>
> Anirudh
>
> On Wed, Nov 21, 2018 at 9:31 AM Marco de Abreu
> <marco.g.abreu@googlemail.com.invalid> wrote:
>
> > Please notice that the "continuous-integration/jenkins/pr-merge"
> currently
> > is overlapping with the new pipelines. Please make sure all checks pass
> > (also the non-required ones) before merging the PRs. I will work on a fix
> > for this overlap.
> >
> > -Marco
> >
> > On Wed, Nov 21, 2018 at 5:42 PM Anton Chernov <mechernov@gmail.com>
> wrote:
> >
> > > The ability to retrigger the pipelines separately is an amazing step
> > > forward. Great job Marco!
> > >
> > > ср, 21 нояб. 2018 г. в 15:03, Marco de Abreu
> > > <marco.g.abreu@googlemail.com.invalid>:
> > >
> > > > Hello,
> > > >
> > > > the PR has been merged and I've created the new pipelines at [1]. You
> > can
> > > > see the new reports if you have a look at this example PR at [2].
> > > >
> > > > The new status messages will be the ones starting with
> > > > "ci/jenkins/mxnet-validation/".
> > > >
> > > > This now allows you to retrigger specific pipelines if they fail. For
> > > > example, if you're interested in the website pipeline, you can now go
> > to
> > > > [3] and just retrigger that instead of running the entire suite.
> > Whenever
> > > > there's a new commit, all pipelines will still be scheduled as before
> > > (the
> > > > overall behaviour or coverage of our pipeline did not change, I just
> > > > decoupled them and increased the usability).
> > > >
> > > > The next step will be the deprecation of the main Jenkinsfile (the
> one
> > > > which reports the status as
> "continuous-integration/jenkins/pr-merge")
> > > and
> > > > requesting these new statuses to be marked as required (protected
> > master
> > > > branch). Since we have to change some reporting tools to point to the
> > new
> > > > jobs and I'd like to observe the stability for some time, this will
> > take
> > > > some times.
> > > >
> > > > You can now resume changes in the Jenkinsfiles. But please do not
> > modify
> > > > the Jenkinsfile in the root directory but instead the ones at [4].
> The
> > > > nightly Jenkinsfiles (or basically all Jenkinsfiles that are not part
> > of
> > > > the main pipeline) have not been migrated yet and I will do that at a
> > > later
> > > > point in time.
> > > >
> > > > Best regards,
> > > > Marco
> > > >
> > > > [1]: http://jenkins.mxnet-ci.amazon-ml.com/job/mxnet-validation/
> > > > [2]: https://github.com/apache/incubator-mxnet/pull/13352
> > > > [3]:
> > > >
> > > >
> > >
> >
> http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fwebsite/detail/PR-13352/1/pipeline
> > > > [4]:
> https://github.com/apache/incubator-mxnet/tree/master/ci/jenkins
> > > >
> > > > On Tue, Nov 20, 2018 at 9:33 PM Marco de Abreu <
> > > > marco.g.abreu@googlemail.com>
> > > > wrote:
> > > >
> > > > > I have just submitted my PR at
> > > > > https://github.com/apache/incubator-mxnet/pull/13344. Test jobs
> are
> > > > > available at
> > > > > http://jenkins.mxnet-ci-dev.amazon-ml.com/view/test-marco-mxnet/.
> > > > >
> > > > > As soon as I'm done with my tests, I will mark it as ready for
> > review.
> > > > >
> > > > > Best regards,
> > > > > Marco
> > > > >
> > > > > On Tue, Nov 20, 2018 at 9:09 PM Marco de Abreu <
> > > > > marco.g.abreu@googlemail.com> wrote:
> > > > >
> > > > >> Thanks, Pedro!
> > > > >>
> > > > >> I have also been looking into that issue, but it seems like this
> > would
> > > > >> require changes in the groovy interpreter of Jenkins. From what
I
> > can
> > > > tell,
> > > > >> a refactor will give us multiple benefits (clarity and speed)
> aside
> > > from
> > > > >> resolving this issue.
> > > > >>
> > > > >> Best regards,
> > > > >> Marco
> > > > >>
> > > > >> Am Di., 20. Nov. 2018, 19:54 hat Pedro Larroy <
> > > > >> pedro.larroy.lists@gmail.com> geschrieben:
> > > > >>
> > > > >>> I think this is a big problem, which has blocked us before.
I
> want
> > to
> > > > >>> point out that you are doing a great thing by avoiding everyone
> > > > >>> getting blocked by refactoring the pipelines.
> > > > >>>
> > > > >>> My concern is that we are kicking the can down the road and
not
> > > > >>> addressing the root cause of the problem with is known
> > > > >>> https://issues.jenkins-ci.org/browse/JENKINS-37984
> > > > >>>
> > > > >>> Pedro.
> > > > >>>
> > > > >>>
> > > > >>> On Tue, Nov 20, 2018 at 6:08 PM Marco de Abreu
> > > > >>> <marco.g.abreu@googlemail.com.invalid> wrote:
> > > > >>> >
> > > > >>> > Hello Steffen,
> > > > >>> >
> > > > >>> > no, there won't be any impact on the PR process or nightly
> > > > regressions.
> > > > >>> > Only the reporting will have to be updated with the
new job
> > links,
> > > > but
> > > > >>> that
> > > > >>> > should be a minor issue. To avoid any outage, I have
been
> > thinking
> > > > >>> about
> > > > >>> > running both versions in parallel.
> > > > >>> >
> > > > >>> > Best regards,
> > > > >>> > Marco
> > > > >>> >
> > > > >>> >
> > > > >>> >
> > > > >>> > On Tue, Nov 20, 2018 at 5:53 PM Steffen Rochel <
> > > > >>> steffenrochel@gmail.com>
> > > > >>> > wrote:
> > > > >>> >
> > > > >>> > > Hi Marco - is there any impact on reporting, the
PR process
> or
> > > > >>> nightly
> > > > >>> > > regression beside reduction in TAT?  If yes, please
> elaborate.
> > > > >>> > > Steffen
> > > > >>> > >
> > > > >>> > > On Tue, Nov 20, 2018 at 8:05 AM Marco de Abreu
> > > > >>> > > <marco.g.abreu@googlemail.com.invalid> wrote:
> > > > >>> > >
> > > > >>> > > > Hello,
> > > > >>> > > >
> > > > >>> > > > we ran into issues around the maximum filesize
of the
> > > Jenkinsfile
> > > > >>> a few
> > > > >>> > > > times already. In order to resolve this issue,
I'd like to
> > > > combine
> > > > >>> this
> > > > >>> > > > with some refactors I have planned for quite
some time.
> > > > >>> > > >
> > > > >>> > > > The idea is basically to move away from one
big Jenkinsfile
> > and
> > > > >>> instead
> > > > >>> > > > split it into separate jobs that run in parallel
and report
> > > their
> > > > >>> status
> > > > >>> > > > individually. Besides avoiding the size restriction,
this
> > will
> > > > >>> greatly
> > > > >>> > > > speed up the PR validation process by reducing
the critical
> > > path.
> > > > >>> Instead
> > > > >>> > > > of having to wait for every single step within
a stage to
> > > finish
> > > > >>> before
> > > > >>> > > the
> > > > >>> > > > next stage (e.g. tests) is getting executed,
these
> pipelines
> > > > would
> > > > >>> now be
> > > > >>> > > > able to move forward individually. I'm still
in the process
> > of
> > > > >>> > > refactoring
> > > > >>> > > > and can't provide any numbers or documentation
at this
> time,
> > > but
> > > > I
> > > > >>> would
> > > > >>> > > > like to announce this early on to avoid conflicts:
> > > > >>> > > >
> > > > >>> > > > Since I will remove the original Jenkinsfile,
this might
> > cause
> > > > >>> conflicts
> > > > >>> > > > with ongoing efforts that try to change the
Jenkinsfile.
> This
> > > > >>> poses the
> > > > >>> > > > risk that I might forget to port a change.
Thus, I'd like
> to
> > > ask
> > > > >>> all
> > > > >>> > > > contributors to wait with changes of Jenkinsfile
and would
> > like
> > > > to
> > > > >>> > > request
> > > > >>> > > > fellow-committers to wait with merging any
> > Jenkinsfile-related
> > > > PRs
> > > > >>> until
> > > > >>> > > > further notice.
> > > > >>> > > >
> > > > >>> > > > I expect to finish this refactor until the
end of the week.
> > > > Please
> > > > >>> don't
> > > > >>> > > > hesitate to ask if you've got further questions.
> > > > >>> > > >
> > > > >>> > > > Please excuse any caused inconveniences.
> > > > >>> > > >
> > > > >>> > > > Best regards,
> > > > >>> > > > Marco
> > > > >>> > > >
> > > > >>> > >
> > > > >>>
> > > > >>
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message