mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marco de Abreu <marco.g.ab...@googlemail.com.INVALID>
Subject Re: Splitting Jenkins pipelines - stop changes to Jenkinsfiles!
Date Tue, 20 Nov 2018 20:33:39 GMT
I have just submitted my PR at
https://github.com/apache/incubator-mxnet/pull/13344. Test jobs are
available at
http://jenkins.mxnet-ci-dev.amazon-ml.com/view/test-marco-mxnet/.

As soon as I'm done with my tests, I will mark it as ready for review.

Best regards,
Marco

On Tue, Nov 20, 2018 at 9:09 PM Marco de Abreu <marco.g.abreu@googlemail.com>
wrote:

> Thanks, Pedro!
>
> I have also been looking into that issue, but it seems like this would
> require changes in the groovy interpreter of Jenkins. From what I can tell,
> a refactor will give us multiple benefits (clarity and speed) aside from
> resolving this issue.
>
> Best regards,
> Marco
>
> Am Di., 20. Nov. 2018, 19:54 hat Pedro Larroy <
> pedro.larroy.lists@gmail.com> geschrieben:
>
>> I think this is a big problem, which has blocked us before. I want to
>> point out that you are doing a great thing by avoiding everyone
>> getting blocked by refactoring the pipelines.
>>
>> My concern is that we are kicking the can down the road and not
>> addressing the root cause of the problem with is known
>> https://issues.jenkins-ci.org/browse/JENKINS-37984
>>
>> Pedro.
>>
>>
>> On Tue, Nov 20, 2018 at 6:08 PM Marco de Abreu
>> <marco.g.abreu@googlemail.com.invalid> wrote:
>> >
>> > Hello Steffen,
>> >
>> > no, there won't be any impact on the PR process or nightly regressions.
>> > Only the reporting will have to be updated with the new job links, but
>> that
>> > should be a minor issue. To avoid any outage, I have been thinking about
>> > running both versions in parallel.
>> >
>> > Best regards,
>> > Marco
>> >
>> >
>> >
>> > On Tue, Nov 20, 2018 at 5:53 PM Steffen Rochel <steffenrochel@gmail.com
>> >
>> > wrote:
>> >
>> > > Hi Marco - is there any impact on reporting, the PR process or nightly
>> > > regression beside reduction in TAT?  If yes, please elaborate.
>> > > Steffen
>> > >
>> > > On Tue, Nov 20, 2018 at 8:05 AM Marco de Abreu
>> > > <marco.g.abreu@googlemail.com.invalid> wrote:
>> > >
>> > > > Hello,
>> > > >
>> > > > we ran into issues around the maximum filesize of the Jenkinsfile
a
>> few
>> > > > times already. In order to resolve this issue, I'd like to combine
>> this
>> > > > with some refactors I have planned for quite some time.
>> > > >
>> > > > The idea is basically to move away from one big Jenkinsfile and
>> instead
>> > > > split it into separate jobs that run in parallel and report their
>> status
>> > > > individually. Besides avoiding the size restriction, this will
>> greatly
>> > > > speed up the PR validation process by reducing the critical path.
>> Instead
>> > > > of having to wait for every single step within a stage to finish
>> before
>> > > the
>> > > > next stage (e.g. tests) is getting executed, these pipelines would
>> now be
>> > > > able to move forward individually. I'm still in the process of
>> > > refactoring
>> > > > and can't provide any numbers or documentation at this time, but I
>> would
>> > > > like to announce this early on to avoid conflicts:
>> > > >
>> > > > Since I will remove the original Jenkinsfile, this might cause
>> conflicts
>> > > > with ongoing efforts that try to change the Jenkinsfile. This poses
>> the
>> > > > risk that I might forget to port a change. Thus, I'd like to ask all
>> > > > contributors to wait with changes of Jenkinsfile and would like to
>> > > request
>> > > > fellow-committers to wait with merging any Jenkinsfile-related PRs
>> until
>> > > > further notice.
>> > > >
>> > > > I expect to finish this refactor until the end of the week. Please
>> don't
>> > > > hesitate to ask if you've got further questions.
>> > > >
>> > > > Please excuse any caused inconveniences.
>> > > >
>> > > > Best regards,
>> > > > Marco
>> > > >
>> > >
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message