mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hen <bay...@apache.org>
Subject Re: [Proposal] Stabilizing Apache MXNet CI build system
Date Wed, 01 Nov 2017 20:07:04 GMT
Random question - can the CI be split such that the Apache CI is doing a
basic set of checks on that hardware, and is hooked to a PR, while there is
a larger "Is trunk good for release?" test that is running periodically
rather than on every PR?

ie: do we need each PR to be run on varied hardware, or can we have this
two tier approach?

Hen

On Fri, Oct 20, 2017 at 1:01 PM, sandeep krishnamurthy <
sandeep.krishna98@gmail.com> wrote:

> Hello all,
>
> I am hereby opening up a discussion thread on how we can stabilize Apache
> MXNet CI build system.
>
> Problems:
>
> ========
>
> Recently, we have seen following issues with Apache MXNet CI build systems:
>
>    1. Apache Jenkins master is overloaded and we see issues like - unable
>    to trigger builds, difficult to load and view the blue ocean and other
>    Jenkins build status page.
>    2. We are generating too many request/interaction on Apache Infra team.
>       1. Addition/deletion of new slave: Caused from scaling activity,
>       recycling, troubleshooting or any actions leading to change of slave
>       machines.
>       2. Plugins / other Jenkins Master configurations.
>       3. Experimentation on CI pipelines.
>    3. Harder to debug and resolve issues - Since access to master and slave
>    is not with the same community, it requires Infra and community to dive
>    deep together on all action items.
>
> Possible Solutions:
>
> ==============
>
>    1. Can we set up a separate Jenkins CI build system for Apache MXNet
>    outside Apache Infra?
>    2. Can we have a separate Jenkins Master in Apache Infra for MXNet?
>    3. Review design of current setup, refine and fill the gaps.
>
> @ Mentors/Infra team/Community:
>
> ==========================
>
> Please provide your suggestions on how we can proceed further and work on
> stabilizing the CI build systems for MXNet.
>
> Also, if the community decides on separate Jenkins CI build system, what
> important points should be taken care of apart from the below:
>
>    1. Community being able to access the build page for build statuses.
>    2. Committers being able to login with apache credentials.
>    3. Hook setup from apache/incubator-mxnet repo to Jenkins master.
>
>
> Irrespective of the solution we come up, I think we should initiate a
> technical design discussion on how to setup the CI build system. Probably 1
> or 2 pager documents with the architecture and review with Infra and
> community members.
>
> ***There were few proposal and discussion on the slack channel, to reach
> wider community members, moving that discussion formally to this list.
>
>
> My Proposal: Option 1 - Set up separate Jenkins CI build system.
>
> Thanks,
>
> Sandeep
>
>
>
> --
> Sandeep Krishnamurthy
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message