mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shiwen hu <yajiedes...@gmail.com>
Subject Re: MXNet -> Apache Migration proposal
Date Sat, 01 Jul 2017 07:14:20 GMT
what problem with windows ci?

2017-07-01 9:06 GMT+08:00 Ly Nguyen <nguyenlyx@gmail.com>:

> This week's summary:
> 1. Wrote FAQ and publicized CI wiki
> 2. Plan was to complete migration by end of next week
>     1. Spent 1.5 days trying to set up Windows slave - was not successful
> and would find it more productive to create an AMI from currently running
> slaves. Mu says a running Windows slave is not necessary for migration but
> that means we would be losing Windows coverage.
>     2. The goal for this week was to ensure that PRs, merges, nightlies
> against the fork trigger builds that pass. There were a lot of hurdles.
> Many items had to happen in sequence and depended on others’ schedules.
> Namely,
>         1. accepted invitation to be committer on Monday morning
>         2. received Apache account Tuesday morning
>         3. got access to Jenkins & repo Wednesday morning
>         4. filed tickets for the Infra team to add webhooks which was
> addressed this morning https://issues.apache.org/jira/browse/INFRA-14472
>         5. Apache builds of all projects including MXNet’s were not
> happening because of some infra issue so there was not much traction today
> https://issues.apache.org/jira/browse/INFRA-14476
>     3. Filed a ticket for support on building docs website
> https://issues.apache.org/jira/browse/INFRA-14479
>     4. Filed a ticket to reconfigure donated linux slaves
> https://issues.apache.org/jira/browse/INFRA-14478
>
> On Tue, Jun 27, 2017 at 1:10 PM, Ly Nguyen <nguyenlyx@gmail.com> wrote:
>
> > We are aiming to complete migration of MXNet to Apache by July 10. This
> > involves transferring the GitHub repo ownership to Apache.
> >
> > Migration is tracked at this project board:
> https://github.com/dmlc/mxnet/
> > projects/6
> > As a part of the migration, we also need to adopt the Apache release
> > process for our next release which is mid-July. This wiki
> > <https://cwiki.apache.org/confluence/display/MXNET/
> Continuous+Integration>
> > gives an overview of of how the process works. It also lists some
> > automation tasks that come after the completion of code base migration
> and
> > the next release.
> >
> > FAQ:
> >
> >    1.
> >
> >    Why are we migrating the code base to Apache ownership?
> >    1.
> >
> >       This is one of the steps on graduating from Apache incubation.
> >       2.
> >
> >    When is this happening?
> >    1.
> >
> >       We are aiming for migration to complete by July 10th.
> >       3.
> >
> >    Will my commits/contributions still exist after migration?
> >    1.
> >
> >       Yes. Existing commits will still appear under your existing github
> >       id, and stats will carry over. New commits will also appear under
> your
> >       existing github id, so long as you’ve configured your ~/.gitconfig
> with an
> >       email address which you’ve linked in your github profile.
> >       2.
> >
> >       Committers will also need to link their Apache ids with the github
> >       ids to gain write access, in which case, the above answer still
> applies.
> >       See #9 on how to link your Apache id.
> >       4.
> >
> >    What will happen to my in flight pull requests?
> >    1.
> >
> >       It will remain intact
> >       5.
> >
> >    Will I still be a member/owner after migration?
> >    1.
> >
> >       Current list of Apache MXNet committers: https://wiki.apache.org/
> >       incubator/MXNetProposal
> >       2.
> >
> >       If you’re not an Apache committer, you lose membership/ownership
> >       rights
> >       3.
> >
> >       Apache Infra are the only people with Owner/Admin permissions there
> >       4.
> >
> >       Apache committers will have write access
> >       6.
> >
> >    What other things will be transferred with the repository?
> >    1.
> >
> >       Wiki, stars, watchers, webhooks, services, deploy keys
> >       7.
> >
> >    What will my fork be associated with after migration?
> >    1.
> >
> >       It will remain associated with the transferred repository
> >       8.
> >
> >    Will I have to change all references to http://github.com/dmlc/mxnet
> ?
> >    1.
> >
> >       All links to http://github.com/dmlc/mxnet will automatically be
> >       redirected to new location when issuing `git clone`, `git fetch`,
> `git
> >       push`, etc, (as long as we don’t create another “mxnet” repository
> under
> >       DMLC). However, to avoid confusion, you can change the links where
> >       possible, and change remote: `git remote set-url origin new_url`
> >       9.
> >
> >    How do I gain write access to the repo?
> >    1.
> >
> >       First, you need to be a committer. Then use
> >       https://gitbox.apache.org/setup/ <https://gitbox.apache.org/setup/
> >
> >       to associate the Apache and GitHub accounts. Note that all
> committers will
> >       need to enable 2-factor authentication on GitHub
> >       10.
> >
> >    Are we also moving mxnet CI? If so, what is the new location? Will
> >    nightly tests continue to run? How can I add new tests?
> >    1.
> >
> >       We will rely on Apache’s build server to run our builds.
> >       2.
> >
> >       It will first only run unit tests for PRs and merges. Tests can be
> >       added following the structure setup in
> >       https://github.com/dmlc/mxnet/blob/master/Jenkinsfile
> >       <https://github.com/dmlc/mxnet/blob/master/Jenkinsfile> .
> >       3.
> >
> >       Nightly tests are currently running at http://jenkins-master-elb-
> >       1979848568.us-east-1.elb.amazonaws.com/
> >       <http://jenkins-master-elb-1979848568.us-east-1.elb.amazonaws.com/
> >
> >       and will gradually run in Apache’s build server too. There, we
> will provide
> >       artifacts such as pip wheels and source packages for the community
> to test.
> >       1.
> >
> >          Automated releases will happen on http://jenkins-master-elb-
> >          1979848568.us-east-1.elb.amazonaws.com/
> >          <http://jenkins-master-elb-1979848568.us-east-1.elb.
> amazonaws.com/>
> >          as Apache’s build doesn’t support key storage.
> >          11.
> >
> >    Is mxnet.io moving too?
> >    1.
> >
> >       For some time we will have both mxnet.apache.org and mxnet.io
> >       hosting the docs. When we are confident that mxnet.apache.org is
> >       stable, we will redirect mxnet.io to there.
> >
> >
> > Link on GitHub repo transfers: https://help.github.com/
> > articles/about-repository-transfers/
> >
> > Feel free to ask any other questions.
> >
> >
> >
> > On Wed, Jun 7, 2017 at 12:53 PM, Ly Nguyen <nguyenlyx@gmail.com> wrote:
> >
> >> I’ve documented the detailed steps below on the process of migrating
> >> MXNet -> Apache for open feedback and discussion.
> >>
> >> Essentially Amazon will be providing the GPU build slaves to be hooked
> >> into Apache’s Jenkins build Master. We’ll first make sure that Apache
> can
> >> build a fork of MXNet, before officially transferring ownership of the
> >> MXNet repo.
> >>
> >> Steps to migration:
> >> 1.      Provide Apache with Linux slaves & slave tags
> >> a.      Provide Apache with slave configuration (tags, remote root dir,
> >> etc.)
> >> b.      Spin up 6 slaves
> >> c.       Launch connection via JNLP
> >> 2.      Apache forks MXNet repo and makes sure builds are successful on
> >> their build set up
> >> a.      Ask Apache to give me committer rights
> >> b.      I remove the Windows jobs until a later time
> >> c.       Apache sets up Jenkins jobs and Github webhooks
> >>                                                               i.
> >> Build every commit and origin/fork PR’s without merge (main Jenkinsfile)
> >>                                                             ii.
> >> Nightly job (nightly Jenkins file, will start with a dummy one and add
> more
> >> configurations later)
> >> d.      If Windows slave setup is available, provide it to Apache and
> >> enable the jobs again
> >> 3.      Transfer the repo and point the build set up there
> >> 4.      Apache deploys the docs to their website
> >>
> >> Open security questions:
> >> 1.      How can we ensure that our slaves are not used by other
> projects?
> >> a.      It’s not, it’s a social contract.
> >> 2.      To protect the slave hosts, would running Jenkins slave inside a
> >> Docker container be a solution, or is there a recommended best practice?
> >> a.      Run slave behind a NAT gateway and launch via JNLP
> >> 3.      Does Apache place SSH key inside the build host for Docs
> >> deployment to the website? Are there security concerns there?
> >> a.      The only slaves that are allowed to deploy docs are
> >> ASF-controlled. Just provide the build command.
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message