mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Naveen Swamy <mnnav...@gmail.com>
Subject Re: [DISCUSS] improve MXNet Scala release process
Date Sat, 28 Jul 2018 00:26:18 GMT
I collaborate with Qing as a part of my day time job, to give you a little
more perspective on the proposed work

For 1)
What we found is that users often run into conflicts when they use a
different version of the dependency(CUDA, CUDNN, OpenBLAS, OpenCV, etc,.)
and the one we build with MXNet backend and use in the MXNet Scala package.
Also it makes its not very straight-forward for users to install these
dependencies themselves in order to lower the entry barrier and to make
everything work out of the box we are thinking to build MXNet all these
dependencies with MXNet (as a static library) and embed them in the MXNet
Scala package. This is also inspired by the work you have done for Apache
MXNet pip packages, Ideally I would like to reuse some of that work.

Maven does not manage the binaries, you still have to build the binary and
release them but what dependencies the binaries are built with is what
causes the confusion and problems.
In the past there were 20+ packages of MXNet and I reduced it to 3 (OSX,
Linux-CPU, Linux-GPU -- please see the discussion thread below), with the
latest version of the dependencies and we'll build/manage additional
packages based on the user demand.

Please see the previous discussion about this topic.

1) Scala Maven Package discussion:
https://lists.apache.org/thread.html/c3846515fc5560d826e7b6f47e9b8b6728a925e6f8decb9676456144@%3Cdev.mxnet.apache.org%3E
2) Config for Scala Package:
https://lists.apache.org/thread.html/0be6beb89cc2a792e7ba861a251f9a9a0b81fa36a5a0cd59d9f2cb6f@%3Cdev.mxnet.apache.org%3E
3) Current Scala Package Release process:
https://cwiki.apache.org/confluence/display/MXNET/MXNet-Scala+Release+Process


Hope that answers


On Fri, Jul 27, 2018 at 4:59 PM, Sheng Zha <szha.pvg@gmail.com> wrote:

> Qing,
>
> For 1, why would it be a blocker, given that there were previous releases?
> Has there been compatibility issues for scala packages? If so, why did we
> release?
> There are many maven packages that include binary already, so if we can
> find the binary for all dependency it's probably best to link to them, and
> let maven manage these dependencies.
>
> For 2, since the current CI is based on Jenkins, I imagine it would be some
> sort of Jenkins pipeline? Other people who're better versed in Jenkins can
> chime in.
>
> Personally, I'm interested in 3 as well. I have a pipeline for building pip
> packages that's currently not utilizing the CI, and the item 3 is the
> blocker too. Once you finish, it would be great to refer to the same
> solution, so that I can move it into the same CI.
>
> -sz
>
> On Fri, Jul 27, 2018 at 4:37 PM, Qing Lan <lanking520@live.com> wrote:
>
> > Hi all,
> >
> > Recently contributors on Scala Language development worked together and
> > finally able to publish Scala package on Maven. Now I would like to
> raise a
> > discussion to automate Scala release process and also discover a standard
> > way to release different packages for MXNet so we won’t ask any
> individuals
> > to spend a long time to publish the package.
> >
> > There are three blocks that stop this automated process:
> >
> >   1.  How to build general hardware-compatible backend dependencies for
> > MXNet (Linux CPU/GPU Mac OSX)
> >   2.  How to automate the frontend release process and CI integration
> >   3.  How to keep credentials for the release in the pipeline
> >
> > Scala Release process created by Naveen: https://cwiki.apache.org/
> > confluence/display/MXNET/MXNet-Scala+Release+Process
> >
> > Thanks,
> > Qing
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message