mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Naveen Swamy <mnnav...@gmail.com>
Subject Re: [DISCUSS] improve MXNet Scala release process
Date Fri, 03 Aug 2018 17:26:00 GMT
Hi Carin,

The thinking right now is to publish nightly to the Apache Snapshot
repository, building and validating with integration tests. We(Qing, me and
Andrew Ayres) will work on a elaborate document detailing the process and
we'll loop you in as well.

Thanks, Naveen

On Fri, Aug 3, 2018 at 8:35 AM, Carin Meier <carinmeier@gmail.com> wrote:

> Hi,
>
> I was thinking about the process for publishing the Clojure jar as well. I
> think, since it will be published to Nexus/Maven and the building of it
> depends on the Scala jar artifact, it might make sense to combine the
> publishing of the Clojure jar at the same time as the Scala Jar. I haven't
> worked out all the details yet, but I'm thinking with that it would be a
> couple command lines in the same script that handles the Scala deployment.
>
> Or if it is a separate process, it might be able to share common parts.
>
> Could you please keep me in the loop of the deployment process so we can
> figure out the best place to work in Clojure as well?
>
> Thanks,
> Carin
>
> On Tue, Jul 31, 2018 at 3:49 PM, Qing Lan <lanking520@live.com> wrote:
>
> > Upon offline discussion with Marco,
> >
> > He proposed a plan that can actually help us conduct 3):
> >         1. This job will not be trigger when PR runs and strictly limit
> > that only committer can run the restricted job.
> >         2. The code being run in there will only covers the code from the
> > branch you choose to go, it will be committers responsibilities not to
> > merge any trivial credential grabber code.
> >         3. Test this is simple. The restricted job uses a similar
> > architecture with current CI. You can send a PR with dockerfiles, scripts
> > and configurations on Jenkins to give it a test to run the job with a
> mock
> > credential. Finally please contact people working on CI to give it a test
> > run and they will do the last step to merge your change to CI.
> >         4. Marco also mentioned the security level of the credentials.
> The
> > credential being used in the AWS Credential services will be assigned
> with
> > an individual IAM role, which only allows to access to the credentials
> that
> > role being assigned to, and used in the restricted job you have set up.
> >
> > I would also like to encourage people in this list  to join the
> > https://cwiki.apache.org/confluence/display/MXNET/
> > MXNet+Berlin+Office+Hours as the people who is working on improving the
> > CI are there ready to help.
> >
> > Thanks,
> > Qing
> >
> >
> > On 7/28/18, 11:44 PM, "Qing Lan" <lanking520@live.com> wrote:
> >
> >     Thanks Marco, Naveen and Sheng's feedback.
> >
> >     About the 1): Scala side will only pack the mxnet binary only and use
> > dynamic links to all the rest dependencies. So indeed it will require
> users
> > to install all deps as the same as the builder platforms version and this
> > will make them hard to use. Let's please collaborate and create a (set
> of)
> > general CI script(s) to install the deps and bring static links to the
> > package.
> >
> >     About 3): it is indeed a general problems for both Scala and Python
> > publish. If there is a good way we can safely store the credentials, we
> can
> > definitely give automated publish a go. And thanks again for Marco's
> option
> > provided below, I think we can make use of the restricted slaves and give
> > it a test run. And to Marco:
> >         1. Will this restricted jobs being triggered in every PR runs or
> > it just depends on where you put it (like I put in nightly it will never
> >    be trigger in PR)? Will there be a potential risk like a PR attack
> > (create a PR to grab credentials)
> >
> >         2. How do we make sure the coding being run there is under
> control
> > and not be changed by anyone?
> >
> >         3. If I want to test this functionality, where is the best place
> > to create the job and make a test run?
> >
> >     Thanks,
> >     Qing
> >
> >
> >
> >     On 7/27/18, 5:44 PM, "Marco de Abreu" <marco.g.abreu@googlemail.com.
> INVALID>
> > wrote:
> >
> >         Hi all,
> >
> >         about the credential management: We already have a solution based
> > on
> >         restricted slaves [1] and AWS secrets manager [2] that is
> generally
> >         classified to generate binaries and handle credentials. It was
> > designed
> >         with continuous deployment in mind, but we haven't tested it in
> > that field
> >         yet.
> >
> >         To properly assess the requirements, it would be great if we have
> > this
> >         security critical part outlined for each release pipeline. We
> > could then
> >         check and see if our existing solution matches all requirements
> or
> > if
> >         further work is necessary.
> >
> >         Best regards,
> >         Marco
> >
> >         [1]
> >         https://cwiki.apache.org/confluence/display/MXNET/
> > Restricted+jobs+and+nodes
> >         [2] https://docs.aws.amazon.com/secretsmanager/latest/
> > userguide/intro.html
> >
> >     On 7/27/18, 5:43 PM, "Sheng Zha" <szha.pvg@gmail.com> wrote:
> >
> >         Thanks, Naveen. Once we have clarity on 3), it should be no
> > problem for
> >         scala to reuse the same solution. For 1), if this is indeed an
> > issue, it
> >         seems that we may have rushed a bit on the scala releases. Are
> > there any
> >         user reports?
> >
> >         -sz
> >
> >         On Fri, Jul 27, 2018 at 5:26 PM, Naveen Swamy <
> mnnaveen@gmail.com>
> > wrote:
> >
> >         > I collaborate with Qing as a part of my day time job, to give
> > you a little
> >         > more perspective on the proposed work
> >         >
> >         > For 1)
> >         > What we found is that users often run into conflicts when they
> > use a
> >         > different version of the dependency(CUDA, CUDNN, OpenBLAS,
> > OpenCV, etc,.)
> >         > and the one we build with MXNet backend and use in the MXNet
> > Scala package.
> >         > Also it makes its not very straight-forward for users to
> install
> > these
> >         > dependencies themselves in order to lower the entry barrier and
> > to make
> >         > everything work out of the box we are thinking to build MXNet
> > all these
> >         > dependencies with MXNet (as a static library) and embed them in
> > the MXNet
> >         > Scala package. This is also inspired by the work you have done
> > for Apache
> >         > MXNet pip packages, Ideally I would like to reuse some of that
> > work.
> >         >
> >         > Maven does not manage the binaries, you still have to build the
> > binary and
> >         > release them but what dependencies the binaries are built with
> > is what
> >         > causes the confusion and problems.
> >         > In the past there were 20+ packages of MXNet and I reduced it
> to
> > 3 (OSX,
> >         > Linux-CPU, Linux-GPU -- please see the discussion thread
> below),
> > with the
> >         > latest version of the dependencies and we'll build/manage
> > additional
> >         > packages based on the user demand.
> >         >
> >         > Please see the previous discussion about this topic.
> >         >
> >         > 1) Scala Maven Package discussion:
> >         > https://lists.apache.org/thread.html/
> > c3846515fc5560d826e7b6f47e9b8b
> >         > 6728a925e6f8decb9676456144@%3Cdev.mxnet.apache.org%3E
> >         > 2) Config for Scala Package:
> >         > https://lists.apache.org/thread.html/
> > 0be6beb89cc2a792e7ba861a251f9a
> >         > 9a0b81fa36a5a0cd59d9f2cb6f@%3Cdev.mxnet.apache.org%3E
> >         > 3) Current Scala Package Release process:
> >         > https://cwiki.apache.org/confluence/display/MXNET/
> >         > MXNet-Scala+Release+Process
> >         >
> >         >
> >         > Hope that answers
> >         >
> >         >
> >         > On Fri, Jul 27, 2018 at 4:59 PM, Sheng Zha <szha.pvg@gmail.com
> >
> > wrote:
> >         >
> >         > > Qing,
> >         > >
> >         > > For 1, why would it be a blocker, given that there were
> > previous
> >         > releases?
> >         > > Has there been compatibility issues for scala packages? If
> so,
> > why did we
> >         > > release?
> >         > > There are many maven packages that include binary already, so
> > if we can
> >         > > find the binary for all dependency it's probably best to link
> > to them,
> >         > and
> >         > > let maven manage these dependencies.
> >         > >
> >         > > For 2, since the current CI is based on Jenkins, I imagine it
> > would be
> >         > some
> >         > > sort of Jenkins pipeline? Other people who're better versed
> in
> > Jenkins
> >         > can
> >         > > chime in.
> >         > >
> >         > > Personally, I'm interested in 3 as well. I have a pipeline
> for
> > building
> >         > pip
> >         > > packages that's currently not utilizing the CI, and the item
> 3
> > is the
> >         > > blocker too. Once you finish, it would be great to refer to
> > the same
> >         > > solution, so that I can move it into the same CI.
> >         > >
> >         > > -sz
> >         > >
> >         > > On Fri, Jul 27, 2018 at 4:37 PM, Qing Lan <
> lanking520@live.com>
> > wrote:
> >         > >
> >         > > > Hi all,
> >         > > >
> >         > > > Recently contributors on Scala Language development worked
> > together and
> >         > > > finally able to publish Scala package on Maven. Now I would
> > like to
> >         > > raise a
> >         > > > discussion to automate Scala release process and also
> > discover a
> >         > standard
> >         > > > way to release different packages for MXNet so we won’t
ask
> > any
> >         > > individuals
> >         > > > to spend a long time to publish the package.
> >         > > >
> >         > > > There are three blocks that stop this automated process:
> >         > > >
> >         > > >   1.  How to build general hardware-compatible backend
> > dependencies for
> >         > > > MXNet (Linux CPU/GPU Mac OSX)
> >         > > >   2.  How to automate the frontend release process and CI
> > integration
> >         > > >   3.  How to keep credentials for the release in the
> pipeline
> >         > > >
> >         > > > Scala Release process created by Naveen:
> > https://cwiki.apache.org/
> >         > > > confluence/display/MXNET/MXNet-Scala+Release+Process
> >         > > >
> >         > > > Thanks,
> >         > > > Qing
> >         > > >
> >         > > >
> >         > >
> >         >
> >
> >
> >
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message