Mailing-List: contact dev-help@mxnet.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@mxnet.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <1499536107890.9256@amazon.com>
References: <CAOKkJGpjbkOCv_8L29btWshR60Rp9m-+T-3nVqsnrVkc6k=H4A@mail.gmail.com>
 <CAOKkJGrjF2Zm7u=XgJB0yevnjirUGLH+HmKO5sSsD0H6B_1czw@mail.gmail.com>
 <CAOKkJGq6ikspYVS7i6HH9E2kaFJ7AU_gaVESgsJ4VsiEm-Ptzw@mail.gmail.com>
 <CAAnCdEBOY+EBUb8EcH4p8a+AqWk0o5MrJWm_FGMPgk3zkGjC5A@mail.gmail.com>
 <CAAy-JxAnh-wH1PRGg95NWt=wnBYdCn4GOeh=Sf3nJEk_m=f-xQ@mail.gmail.com>
 <CAAnCdEDKks-+H_Cf+Xzw7cSA9etynJLnokNYVXek8ewtdrpJJA@mail.gmail.com>
 <CAOKkJGotW8NuPz-NW9NWhOt70GckQ+m=B70+LadAAP3Uz6SO3g@mail.gmail.com>
 <CAFBTSW+cQHY6PB3xjdCD180npfeNY-oGG6G0xvM3y1r7t-LC=Q@mail.gmail.com>
 <CAKYPbiuu0VNvTpnCm95sBo5UHaSBsvW8kYvcoa4=Uw5BufmR3A@mail.gmail.com> <1499536107890.9256@amazon.com>
From: Suneel Marthi <smarthi@apache.org>
Date: Sat, 8 Jul 2017 14:33:00 -0400
Message-ID: <CAOtpBjjZUpwq90oY_aGss_s9qH-7jXhfYsqXrbOJwkeEht=zPQ@mail.gmail.com>
Subject: Re: MXNet -> Apache Migration proposal
To: dev@mxnet.incubator.apache.org
Content-Type: multipart/alternative; boundary="94eb2c09421662f5fd0553d29484"
archived-at: Sat, 08 Jul 2017 18:33:16 -0000

--94eb2c09421662f5fd0553d29484
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

I suppose we'll be migrating to apache Gitbox, correct ??

On Sat, Jul 8, 2017 at 1:48 PM, Li, Mu <mli@amazon.com> wrote:

> Can we schedule a time on monday to finish the transfer together?
> ________________________________________
> From: Daniel Pono Takamori <pono@apache.org>
> Sent: Friday, July 7, 2017 5:22 PM
> To: dev@mxnet.incubator.apache.org
> Subject: Re: MXNet -> Apache Migration proposal
>
> This looks great Ly.  On Monday we can transfer the main repo over and
> then you can change the builds to point to the official repo.  I'll
> make a clone of the website next week so we can do a similar thing to
> test building the website and then we can publish it to
> mxnet.incubator.apache.org.  Once we have that worked out we can move
> forward with transferring the domain to our control.
>
> Have a great weekend everyone!
>
> On Fri, Jul 7, 2017 at 6:07 PM, Dominic Divakaruni
> <dominic.divakaruni@gmail.com> wrote:
> > great stuff!! glad to see this getting close!
> >
> > On Fri, Jul 7, 2017 at 3:47 PM, Ly Nguyen <nguyenlyx@gmail.com> wrote:
> >>
> >> We have successfully validated that merges and pull requests against a=
n
> >> Apache fork of MXNet runs successfully on builds.apache.org:
> >>
> >> https://builds.apache.org/blue/organizations/jenkins/
> incubator-mxnet-master2/detail/master/13/pipeline
> >>
> >> https://builds.apache.org/blue/organizations/jenkins/
> incubator-mxnet-master2/detail/PR-3/1/pipeline
> >>
> >> We have also added a dummy nightly run to be populated with builds and
> >> test
> >> cases after migration, as discussed. We can now move forward with the
> >> migration to Apache and I recommend the following steps:
> >> - [ ] Add Pono as owner
> >> - [ ] Pono adds Apache git hooks to MXNet repo
> >> - [ ] Change source control of Apache Jenkins jobs to point to MXNet
> repo,
> >> verify a run is successful
> >> - [ ] Change MXNet org to Apache, verify a run is successful, mxnet.io
> >> still building
> >> - [ ] Start docs build to mxnet.apache.org
> >> Note that one kink to iron out is that PR build statuses aren=E2=80=99=
t being
> >> updated. Here=E2=80=99s a ticket to follow:
> >>
> >> https://issues.apache.org/jira/secure/RapidBoard.jspa?
> rapidView=3D25&projectKey=3DINFRA&view=3Ddetail&selectedIssue=3DINFRA-145=
40
> >>
> >>
> >> On Sat, Jul 1, 2017 at 9:15 PM, shiwen hu <yajiedesign@gmail.com>
> wrote:
> >>
> >> > 1. The `mxnet directory` is a directory on the current CI server. He
> >> > contains the necessary files, including library dependencies, data
> files
> >> > needed for testing, compiling scripts needed, and so on.You can find
> Mu
> >> > Li
> >> > and ask him to copy from the current Ci
> >> > 2.Graphics Driver downlaod from
> >> > http://www.nvidia.com/Download/index.aspx
> >> > 3.Luanch is a small program. As long as you run it, you should be ab=
le
> >> > to
> >> > see what to do at a glance
> >> >
> >> > 2017-07-02 10:03 GMT+08:00 Naveen Swamy <mnnaveen@gmail.com>:
> >> >
> >> > > @yajiedesign
> >> > > we are building a new slave to be used in Apache Infra, the
> >> > > instructions
> >> > > here
> >> > > https://gist.github.com/yajiedesign/40b3809b51a1706d353e9129071b14
> >> > fb
> >> > > to setup a new slave from scratch is insufficient(probably
> outdated),
> >> > > we
> >> > > ran into quite a bit of problem setting up OpenBlas and OpenCV(tho=
se
> >> > > instructions were missing) though we were able to get through thes=
e
> >> > > problems we anticipate further problems.
> >> > > since we want to move our Infrastructure to Apache by the end of
> next
> >> > week
> >> > > we have paused the effort of setting up a Windows slave and testin=
g
> >> > > the
> >> > > Linux slaves that are already setup.
> >> > >
> >> > > Is it possible for you to update those instructions? meanwhile, we
> >> > > have
> >> > > requested Mu Li to create an AMI out of the existing slave.
> >> > >
> >> > > Can I also request you to provide instructions on how to create pi=
p
> >> > package
> >> > > for Windows? currently, 0.10 version does not have windows pip
> >> > > package?
> >> > >
> >> > > Thanks, Naveen
> >> > >
> >> > >
> >> > >
> >> > > On Sat, Jul 1, 2017 at 12:14 AM, shiwen hu <yajiedesign@gmail.com>
> >> > wrote:
> >> > >
> >> > > > what problem with windows ci?
> >> > > >
> >> > > > 2017-07-01 9:06 GMT+08:00 Ly Nguyen <nguyenlyx@gmail.com>:
> >> > > >
> >> > > > > This week's summary:
> >> > > > > 1. Wrote FAQ and publicized CI wiki
> >> > > > > 2. Plan was to complete migration by end of next week
> >> > > > >     1. Spent 1.5 days trying to set up Windows slave - was not
> >> > > successful
> >> > > > > and would find it more productive to create an AMI from
> currently
> >> > > running
> >> > > > > slaves. Mu says a running Windows slave is not necessary for
> >> > migration
> >> > > > but
> >> > > > > that means we would be losing Windows coverage.
> >> > > > >     2. The goal for this week was to ensure that PRs, merges,
> >> > nightlies
> >> > > > > against the fork trigger builds that pass. There were a lot of
> >> > hurdles.
> >> > > > > Many items had to happen in sequence and depended on others=E2=
=80=99
> >> > schedules.
> >> > > > > Namely,
> >> > > > >         1. accepted invitation to be committer on Monday morni=
ng
> >> > > > >         2. received Apache account Tuesday morning
> >> > > > >         3. got access to Jenkins & repo Wednesday morning
> >> > > > >         4. filed tickets for the Infra team to add webhooks
> which
> >> > > > > was
> >> > > > > addressed this morning https://issues.apache.org/
> >> > > jira/browse/INFRA-14472
> >> > > > >         5. Apache builds of all projects including MXNet=E2=80=
=99s were
> >> > > > > not
> >> > > > > happening because of some infra issue so there was not much
> >> > > > > traction
> >> > > > today
> >> > > > > https://issues.apache.org/jira/browse/INFRA-14476
> >> > > > >     3. Filed a ticket for support on building docs website
> >> > > > > https://issues.apache.org/jira/browse/INFRA-14479
> >> > > > >     4. Filed a ticket to reconfigure donated linux slaves
> >> > > > > https://issues.apache.org/jira/browse/INFRA-14478
> >> > > > >
> >> > > > > On Tue, Jun 27, 2017 at 1:10 PM, Ly Nguyen <nguyenlyx@gmail.co=
m
> >
> >> > > wrote:
> >> > > > >
> >> > > > > > We are aiming to complete migration of MXNet to Apache by Ju=
ly
> >> > > > > > 10.
> >> > > This
> >> > > > > > involves transferring the GitHub repo ownership to Apache.
> >> > > > > >
> >> > > > > > Migration is tracked at this project board:
> >> > > > > https://github.com/dmlc/mxnet/
> >> > > > > > projects/6
> >> > > > > > As a part of the migration, we also need to adopt the Apache
> >> > release
> >> > > > > > process for our next release which is mid-July. This wiki
> >> > > > > > <https://cwiki.apache.org/confluence/display/MXNET/
> >> > > > > Continuous+Integration>
> >> > > > > > gives an overview of of how the process works. It also lists
> >> > > > > > some
> >> > > > > > automation tasks that come after the completion of code base
> >> > > migration
> >> > > > > and
> >> > > > > > the next release.
> >> > > > > >
> >> > > > > > FAQ:
> >> > > > > >
> >> > > > > >    1.
> >> > > > > >
> >> > > > > >    Why are we migrating the code base to Apache ownership?
> >> > > > > >    1.
> >> > > > > >
> >> > > > > >       This is one of the steps on graduating from Apache
> >> > incubation.
> >> > > > > >       2.
> >> > > > > >
> >> > > > > >    When is this happening?
> >> > > > > >    1.
> >> > > > > >
> >> > > > > >       We are aiming for migration to complete by July 10th.
> >> > > > > >       3.
> >> > > > > >
> >> > > > > >    Will my commits/contributions still exist after migration=
?
> >> > > > > >    1.
> >> > > > > >
> >> > > > > >       Yes. Existing commits will still appear under your
> >> > > > > > existing
> >> > > > github
> >> > > > > >       id, and stats will carry over. New commits will also
> >> > > > > > appear
> >> > > under
> >> > > > > your
> >> > > > > >       existing github id, so long as you=E2=80=99ve configur=
ed your
> >> > > > ~/.gitconfig
> >> > > > > with an
> >> > > > > >       email address which you=E2=80=99ve linked in your gith=
ub
> profile.
> >> > > > > >       2.
> >> > > > > >
> >> > > > > >       Committers will also need to link their Apache ids wit=
h
> >> > > > > > the
> >> > > > github
> >> > > > > >       ids to gain write access, in which case, the above
> answer
> >> > still
> >> > > > > applies.
> >> > > > > >       See #9 on how to link your Apache id.
> >> > > > > >       4.
> >> > > > > >
> >> > > > > >    What will happen to my in flight pull requests?
> >> > > > > >    1.
> >> > > > > >
> >> > > > > >       It will remain intact
> >> > > > > >       5.
> >> > > > > >
> >> > > > > >    Will I still be a member/owner after migration?
> >> > > > > >    1.
> >> > > > > >
> >> > > > > >       Current list of Apache MXNet committers:
> >> > > > https://wiki.apache.org/
> >> > > > > >       incubator/MXNetProposal
> >> > > > > >       2.
> >> > > > > >
> >> > > > > >       If you=E2=80=99re not an Apache committer, you lose
> >> > > membership/ownership
> >> > > > > >       rights
> >> > > > > >       3.
> >> > > > > >
> >> > > > > >       Apache Infra are the only people with Owner/Admin
> >> > > > > > permissions
> >> > > > there
> >> > > > > >       4.
> >> > > > > >
> >> > > > > >       Apache committers will have write access
> >> > > > > >       6.
> >> > > > > >
> >> > > > > >    What other things will be transferred with the repository=
?
> >> > > > > >    1.
> >> > > > > >
> >> > > > > >       Wiki, stars, watchers, webhooks, services, deploy keys
> >> > > > > >       7.
> >> > > > > >
> >> > > > > >    What will my fork be associated with after migration?
> >> > > > > >    1.
> >> > > > > >
> >> > > > > >       It will remain associated with the transferred
> repository
> >> > > > > >       8.
> >> > > > > >
> >> > > > > >    Will I have to change all references to
> >> > > > http://github.com/dmlc/mxnet
> >> > > > > ?
> >> > > > > >    1.
> >> > > > > >
> >> > > > > >       All links to http://github.com/dmlc/mxnet will
> >> > > > > > automatically
> >> > > be
> >> > > > > >       redirected to new location when issuing `git clone`,
> `git
> >> > > fetch`,
> >> > > > > `git
> >> > > > > >       push`, etc, (as long as we don=E2=80=99t create anothe=
r =E2=80=9Cmxnet=E2=80=9D
> >> > > > repository
> >> > > > > under
> >> > > > > >       DMLC). However, to avoid confusion, you can change the
> >> > > > > > links
> >> > > > where
> >> > > > > >       possible, and change remote: `git remote set-url origi=
n
> >> > > new_url`
> >> > > > > >       9.
> >> > > > > >
> >> > > > > >    How do I gain write access to the repo?
> >> > > > > >    1.
> >> > > > > >
> >> > > > > >       First, you need to be a committer. Then use
> >> > > > > >       https://gitbox.apache.org/setup/
> >> > > > > > <https://gitbox.apache.org/
> >> > > > setup/
> >> > > > > >
> >> > > > > >       to associate the Apache and GitHub accounts. Note that
> all
> >> > > > > committers will
> >> > > > > >       need to enable 2-factor authentication on GitHub
> >> > > > > >       10.
> >> > > > > >
> >> > > > > >    Are we also moving mxnet CI? If so, what is the new
> location?
> >> > Will
> >> > > > > >    nightly tests continue to run? How can I add new tests?
> >> > > > > >    1.
> >> > > > > >
> >> > > > > >       We will rely on Apache=E2=80=99s build server to run o=
ur builds.
> >> > > > > >       2.
> >> > > > > >
> >> > > > > >       It will first only run unit tests for PRs and merges.
> >> > > > > > Tests
> >> > can
> >> > > > be
> >> > > > > >       added following the structure setup in
> >> > > > > >       https://github.com/dmlc/mxnet/blob/master/Jenkinsfile
> >> > > > > >       <https://github.com/dmlc/mxnet/blob/master/Jenkinsfile=
>
> .
> >> > > > > >       3.
> >> > > > > >
> >> > > > > >       Nightly tests are currently running at
> >> > > > http://jenkins-master-elb-
> >> > > > > >       1979848568.us-east-1.elb.amazonaws.com/
> >> > > > > >       <http://jenkins-master-elb-1979848568.us-east-1.elb.
> >> > > > amazonaws.com/
> >> > > > > >
> >> > > > > >       and will gradually run in Apache=E2=80=99s build serve=
r too.
> >> > > > > > There,
> >> > we
> >> > > > > will provide
> >> > > > > >       artifacts such as pip wheels and source packages for t=
he
> >> > > > community
> >> > > > > to test.
> >> > > > > >       1.
> >> > > > > >
> >> > > > > >          Automated releases will happen on
> >> > > http://jenkins-master-elb-
> >> > > > > >          1979848568.us-east-1.elb.amazonaws.com/
> >> > > > > >          <http://jenkins-master-elb-1979848568.us-east-1.elb=
.
> >> > > > > amazonaws.com/>
> >> > > > > >          as Apache=E2=80=99s build doesn=E2=80=99t support k=
ey storage.
> >> > > > > >          11.
> >> > > > > >
> >> > > > > >    Is mxnet.io moving too?
> >> > > > > >    1.
> >> > > > > >
> >> > > > > >       For some time we will have both mxnet.apache.org and
> >> > mxnet.io
> >> > > > > >       hosting the docs. When we are confident that
> >> > mxnet.apache.org
> >> > > is
> >> > > > > >       stable, we will redirect mxnet.io to there.
> >> > > > > >
> >> > > > > >
> >> > > > > > Link on GitHub repo transfers: https://help.github.com/
> >> > > > > > articles/about-repository-transfers/
> >> > > > > >
> >> > > > > > Feel free to ask any other questions.
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > On Wed, Jun 7, 2017 at 12:53 PM, Ly Nguyen <
> nguyenlyx@gmail.com>
> >> > > > wrote:
> >> > > > > >
> >> > > > > >> I=E2=80=99ve documented the detailed steps below on the pro=
cess of
> >> > migrating
> >> > > > > >> MXNet -> Apache for open feedback and discussion.
> >> > > > > >>
> >> > > > > >> Essentially Amazon will be providing the GPU build slaves t=
o
> be
> >> > > hooked
> >> > > > > >> into Apache=E2=80=99s Jenkins build Master. We=E2=80=99ll f=
irst make sure
> that
> >> > > Apache
> >> > > > > can
> >> > > > > >> build a fork of MXNet, before officially transferring
> ownership
> >> > > > > >> of
> >> > > the
> >> > > > > >> MXNet repo.
> >> > > > > >>
> >> > > > > >> Steps to migration:
> >> > > > > >> 1.      Provide Apache with Linux slaves & slave tags
> >> > > > > >> a.      Provide Apache with slave configuration (tags, remo=
te
> >> > > > > >> root
> >> > > > dir,
> >> > > > > >> etc.)
> >> > > > > >> b.      Spin up 6 slaves
> >> > > > > >> c.       Launch connection via JNLP
> >> > > > > >> 2.      Apache forks MXNet repo and makes sure builds are
> >> > successful
> >> > > > on
> >> > > > > >> their build set up
> >> > > > > >> a.      Ask Apache to give me committer rights
> >> > > > > >> b.      I remove the Windows jobs until a later time
> >> > > > > >> c.       Apache sets up Jenkins jobs and Github webhooks
> >> > > > > >>
> >> > > > > >> i.
> >> > > > > >> Build every commit and origin/fork PR=E2=80=99s without mer=
ge (main
> >> > > > Jenkinsfile)
> >> > > > > >>
>  ii.
> >> > > > > >> Nightly job (nightly Jenkins file, will start with a dummy
> one
> >> > > > > >> and
> >> > > add
> >> > > > > more
> >> > > > > >> configurations later)
> >> > > > > >> d.      If Windows slave setup is available, provide it to
> >> > > > > >> Apache
> >> > > and
> >> > > > > >> enable the jobs again
> >> > > > > >> 3.      Transfer the repo and point the build set up there
> >> > > > > >> 4.      Apache deploys the docs to their website
> >> > > > > >>
> >> > > > > >> Open security questions:
> >> > > > > >> 1.      How can we ensure that our slaves are not used by
> other
> >> > > > > projects?
> >> > > > > >> a.      It=E2=80=99s not, it=E2=80=99s a social contract.
> >> > > > > >> 2.      To protect the slave hosts, would running Jenkins
> slave
> >> > > > inside a
> >> > > > > >> Docker container be a solution, or is there a recommended
> best
> >> > > > practice?
> >> > > > > >> a.      Run slave behind a NAT gateway and launch via JNLP
> >> > > > > >> 3.      Does Apache place SSH key inside the build host for
> >> > > > > >> Docs
> >> > > > > >> deployment to the website? Are there security concerns ther=
e?
> >> > > > > >> a.      The only slaves that are allowed to deploy docs are
> >> > > > > >> ASF-controlled. Just provide the build command.
> >> > > > > >>
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >
> >
> >
> >
> > --
> >
> >
> > Dominic Divakaruni
> > 206.475.9200 Cell
>

--94eb2c09421662f5fd0553d29484--