mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ly Nguyen <nguyen...@gmail.com>
Subject Re: MXNet -> Apache Migration proposal
Date Sat, 01 Jul 2017 01:06:38 GMT
This week's summary:
1. Wrote FAQ and publicized CI wiki
2. Plan was to complete migration by end of next week
    1. Spent 1.5 days trying to set up Windows slave - was not successful
and would find it more productive to create an AMI from currently running
slaves. Mu says a running Windows slave is not necessary for migration but
that means we would be losing Windows coverage.
    2. The goal for this week was to ensure that PRs, merges, nightlies
against the fork trigger builds that pass. There were a lot of hurdles.
Many items had to happen in sequence and depended on others’ schedules.
Namely,
        1. accepted invitation to be committer on Monday morning
        2. received Apache account Tuesday morning
        3. got access to Jenkins & repo Wednesday morning
        4. filed tickets for the Infra team to add webhooks which was
addressed this morning https://issues.apache.org/jira/browse/INFRA-14472
        5. Apache builds of all projects including MXNet’s were not
happening because of some infra issue so there was not much traction today
https://issues.apache.org/jira/browse/INFRA-14476
    3. Filed a ticket for support on building docs website
https://issues.apache.org/jira/browse/INFRA-14479
    4. Filed a ticket to reconfigure donated linux slaves
https://issues.apache.org/jira/browse/INFRA-14478

On Tue, Jun 27, 2017 at 1:10 PM, Ly Nguyen <nguyenlyx@gmail.com> wrote:

> We are aiming to complete migration of MXNet to Apache by July 10. This
> involves transferring the GitHub repo ownership to Apache.
>
> Migration is tracked at this project board: https://github.com/dmlc/mxnet/
> projects/6
> As a part of the migration, we also need to adopt the Apache release
> process for our next release which is mid-July. This wiki
> <https://cwiki.apache.org/confluence/display/MXNET/Continuous+Integration>
> gives an overview of of how the process works. It also lists some
> automation tasks that come after the completion of code base migration and
> the next release.
>
> FAQ:
>
>    1.
>
>    Why are we migrating the code base to Apache ownership?
>    1.
>
>       This is one of the steps on graduating from Apache incubation.
>       2.
>
>    When is this happening?
>    1.
>
>       We are aiming for migration to complete by July 10th.
>       3.
>
>    Will my commits/contributions still exist after migration?
>    1.
>
>       Yes. Existing commits will still appear under your existing github
>       id, and stats will carry over. New commits will also appear under your
>       existing github id, so long as you’ve configured your ~/.gitconfig with an
>       email address which you’ve linked in your github profile.
>       2.
>
>       Committers will also need to link their Apache ids with the github
>       ids to gain write access, in which case, the above answer still applies.
>       See #9 on how to link your Apache id.
>       4.
>
>    What will happen to my in flight pull requests?
>    1.
>
>       It will remain intact
>       5.
>
>    Will I still be a member/owner after migration?
>    1.
>
>       Current list of Apache MXNet committers: https://wiki.apache.org/
>       incubator/MXNetProposal
>       2.
>
>       If you’re not an Apache committer, you lose membership/ownership
>       rights
>       3.
>
>       Apache Infra are the only people with Owner/Admin permissions there
>       4.
>
>       Apache committers will have write access
>       6.
>
>    What other things will be transferred with the repository?
>    1.
>
>       Wiki, stars, watchers, webhooks, services, deploy keys
>       7.
>
>    What will my fork be associated with after migration?
>    1.
>
>       It will remain associated with the transferred repository
>       8.
>
>    Will I have to change all references to http://github.com/dmlc/mxnet ?
>    1.
>
>       All links to http://github.com/dmlc/mxnet will automatically be
>       redirected to new location when issuing `git clone`, `git fetch`, `git
>       push`, etc, (as long as we don’t create another “mxnet” repository under
>       DMLC). However, to avoid confusion, you can change the links where
>       possible, and change remote: `git remote set-url origin new_url`
>       9.
>
>    How do I gain write access to the repo?
>    1.
>
>       First, you need to be a committer. Then use
>       https://gitbox.apache.org/setup/ <https://gitbox.apache.org/setup/>
>       to associate the Apache and GitHub accounts. Note that all committers will
>       need to enable 2-factor authentication on GitHub
>       10.
>
>    Are we also moving mxnet CI? If so, what is the new location? Will
>    nightly tests continue to run? How can I add new tests?
>    1.
>
>       We will rely on Apache’s build server to run our builds.
>       2.
>
>       It will first only run unit tests for PRs and merges. Tests can be
>       added following the structure setup in
>       https://github.com/dmlc/mxnet/blob/master/Jenkinsfile
>       <https://github.com/dmlc/mxnet/blob/master/Jenkinsfile> .
>       3.
>
>       Nightly tests are currently running at http://jenkins-master-elb-
>       1979848568.us-east-1.elb.amazonaws.com/
>       <http://jenkins-master-elb-1979848568.us-east-1.elb.amazonaws.com/>
>       and will gradually run in Apache’s build server too. There, we will provide
>       artifacts such as pip wheels and source packages for the community to test.
>       1.
>
>          Automated releases will happen on http://jenkins-master-elb-
>          1979848568.us-east-1.elb.amazonaws.com/
>          <http://jenkins-master-elb-1979848568.us-east-1.elb.amazonaws.com/>
>          as Apache’s build doesn’t support key storage.
>          11.
>
>    Is mxnet.io moving too?
>    1.
>
>       For some time we will have both mxnet.apache.org and mxnet.io
>       hosting the docs. When we are confident that mxnet.apache.org is
>       stable, we will redirect mxnet.io to there.
>
>
> Link on GitHub repo transfers: https://help.github.com/
> articles/about-repository-transfers/
>
> Feel free to ask any other questions.
>
>
>
> On Wed, Jun 7, 2017 at 12:53 PM, Ly Nguyen <nguyenlyx@gmail.com> wrote:
>
>> I’ve documented the detailed steps below on the process of migrating
>> MXNet -> Apache for open feedback and discussion.
>>
>> Essentially Amazon will be providing the GPU build slaves to be hooked
>> into Apache’s Jenkins build Master. We’ll first make sure that Apache can
>> build a fork of MXNet, before officially transferring ownership of the
>> MXNet repo.
>>
>> Steps to migration:
>> 1.      Provide Apache with Linux slaves & slave tags
>> a.      Provide Apache with slave configuration (tags, remote root dir,
>> etc.)
>> b.      Spin up 6 slaves
>> c.       Launch connection via JNLP
>> 2.      Apache forks MXNet repo and makes sure builds are successful on
>> their build set up
>> a.      Ask Apache to give me committer rights
>> b.      I remove the Windows jobs until a later time
>> c.       Apache sets up Jenkins jobs and Github webhooks
>>                                                               i.
>> Build every commit and origin/fork PR’s without merge (main Jenkinsfile)
>>                                                             ii.
>> Nightly job (nightly Jenkins file, will start with a dummy one and add more
>> configurations later)
>> d.      If Windows slave setup is available, provide it to Apache and
>> enable the jobs again
>> 3.      Transfer the repo and point the build set up there
>> 4.      Apache deploys the docs to their website
>>
>> Open security questions:
>> 1.      How can we ensure that our slaves are not used by other projects?
>> a.      It’s not, it’s a social contract.
>> 2.      To protect the slave hosts, would running Jenkins slave inside a
>> Docker container be a solution, or is there a recommended best practice?
>> a.      Run slave behind a NAT gateway and launch via JNLP
>> 3.      Does Apache place SSH key inside the build host for Docs
>> deployment to the website? Are there security concerns there?
>> a.      The only slaves that are allowed to deploy docs are
>> ASF-controlled. Just provide the build command.
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message