mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Olivier <cjolivie...@gmail.com>
Subject Re: Java API for MXNet
Date Wed, 16 Aug 2017 16:53:47 GMT
+1 for using Swig or JavaCPP to generate a low-level API and then
high-level Scala and Java API's can converge to use that layer.  Scala
diverging towards scripting and Java towards some other API (i.e.
Deeplearning4j, which I have never used).  Using a generated interface
layer makes it much more maintainable IMHO.

On Wed, Aug 16, 2017 at 9:19 AM, Nan Zhu <zhunanmcgill@gmail.com> wrote:

> Hi, Yizhi,
>
> You mean users have 2.10 env while we assemble 2.11 in it?
>
> Best,
>
> Nan
>
> On Wed, Aug 16, 2017 at 9:08 AM, YiZhi Liu <javelinjs@gmail.com> wrote:
>
> > Hi Joern,
> >
> > The point is that, the front is not a simple wrapper of c_api.h, as
> > you mentioned, which can be easily achieved by JavaCPP.
> >
> > I have noticed the potential conflicts between the assembled scala
> > library and the one in users' environment. Can we remove the scala
> > library from the assembly jar? @Nan It wouldn't be a problem since the
> > scala libraries with same major version are compatible.
> >
> > 2017-08-16 23:49 GMT+08:00 Joern Kottmann <kottmann@gmail.com>:
> > > Hello,
> > >
> > > I personally had quite some issues with Scala dependencies in
> > > different versions and Spark, where one version is not compatible with
> > > the other version. Then you need to debug the dependency tree to find
> > > the places where the versions don't match. Every project which would
> > > like to use MXnet then has to depend on Scala and might also get
> > > conflicts if other dependencies depend on different Scala versions.
> > > Probably something which will cause issues for some of your users.
> > > Users who want to use Java might not be familiar with Scala dependency
> > > problems and have a hard time resolving them by getting strange error
> > > messages.
> > >
> > > The JNI layer could be generated with JavaCPP, then we would not need
> > > to write/maintain the C and the  jvm side for that our self.
> > > A good example of JavaCPP and Scala usage is Apache Mahout [1].
> > >
> > > Even if we don't use JavaCPP, the JNI layer should be easy to get into
> > > a state where both can share it, the current Scala JNI layers LibInfo
> > > classes could be converted to Java classes and would in most cases
> > > require only minor changes in the Scala code.
> > >
> > > Jörn
> > >
> > > [1] https://github.com/apache/mahout/tree/master/viennacl/src/main
> > >
> > > On Wed, Aug 16, 2017 at 5:30 PM, Nan Zhu <zhunanmcgill@gmail.com>
> wrote:
> > >> I agree with Yizhi
> > >>
> > >> My major concern is the duplicate implementations, which are usually
> > one of
> > >> the major sources of bugs, especially with two languages which are
> > >> naturally interactive (OK, Calling Scala from Java might need some
> more
> > >> efforts). It is just like we provide C++ & C APIs of MxNet in two
> > separated
> > >> packages.
> > >>
> > >> About dependency problem, when you say "As far as I see this has the
> > great
> > >> disadvantage that the Java API would force Scala as a dependency onto
> > the
> > >> java users.", would you please give a concrete example causing
> critical
> > >> issues?
> > >>
> > >> Best,
> > >>
> > >> Nan
> > >>
> > >>
> > >>
> > >> On Wed, Aug 16, 2017 at 8:19 AM, YiZhi Liu <javelinjs@gmail.com>
> wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> If we build the Java API from the very beginning, i.e. the JNI part,
> > >>> we have to rewrite the codes for training, predict, inferShape, etc.
> > >>> It would be too heavy to maintain a totally new front language.
> > >>>
> > >>> As far as I see, I don't think Scala library dependency would be a
> big
> > >>> problem in most cases, unless we are going to use it in embedded
> > >>> devices. Could you illustrate some use-cases where you cannot involve
> > >>> Scala dependencies?
> > >>>
> > >>> 2017-08-16 22:13 GMT+08:00 Joern Kottmann <kottmann@gmail.com>:
> > >>> > Hello,
> > >>> >
> > >>> > the approach which is taken by Spark is described here [1].
> > >>> >
> > >>> > As far as I see this has the great disadvantage that the Java
API
> > >>> > would force Scala as a dependency onto the java users.
> > >>> > For a library it is always a great advantage if it doesn't have
> many
> > >>> > dependencies, or zero dependencies. In our case it could be quite
> > >>> > realistic to have a thin wrapper around the C API without needing
> any
> > >>> > other dependencies (or only dependencies which can't be avoided).
> > >>> >
> > >>> > The JNI layer could easily be shared between the Java and Scala
> API.
> > >>> > As far as I understand is the JNI layer in the Scala API anyway
> > >>> > private and a change to it wouldn't require that the public part
of
> > >>> > the Scala API is changed.
> > >>> >
> > >>> > What do you think?
> > >>> >
> > >>> > Jörn
> > >>> >
> > >>> > [1] https://cwiki.apache.org/confluence/display/SPARK/Java+
> > API+Internals
> > >>> >
> > >>> > On Wed, Aug 16, 2017 at 3:39 PM, YiZhi Liu <javelinjs@gmail.com>
> > wrote:
> > >>> >> Hi Joern,
> > >>> >>
> > >>> >> I suggest to build Java API as a wrapper of Scala API, re-use
most
> > of
> > >>> >> the procedures. Referring to the Java API in Apache Spark.
> > >>> >>
> > >>> >> 2017-08-16 18:21 GMT+08:00 Joern Kottmann <joern@apache.org>:
> > >>> >>> Hello all,
> > >>> >>>
> > >>> >>> I would like to propose the addition of a Java API to
MXNet.
> > >>> >>>
> > >>> >>> There has been some previous work done for the Scala API,
and it
> > makes
> > >>> >>> sense to at least share the JNI layer between the two.
> > >>> >>>
> > >>> >>> The Java  API probably should be aligned with the Python
API (and
> > >>> >>> others which exist already) with a few changes to give
it a
> native
> > >>> >>> Java feel.
> > >>> >>>
> > >>> >>> As far as I understand there are multiple people interested
to
> > work on
> > >>> >>> this and it would be good to maybe come up with a written
> proposal
> > on
> > >>> >>> how things should be.
> > >>> >>>
> > >>> >>> My motivation is to get a Java API which can be used by
Apache
> > OpenNLP
> > >>> >>> to solve various NLP tasks using Deep Learning based approaches
> > and I
> > >>> >>> am also interested to work on MXNet.
> > >>> >>>
> > >>> >>> Jörn
> > >>> >>
> > >>> >>
> > >>> >>
> > >>> >> --
> > >>> >> Yizhi Liu
> > >>> >> DMLC member
> > >>> >> Technical Manager
> > >>> >> Qihoo 360 Inc, Shanghai, China
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Yizhi Liu
> > >>> DMLC member
> > >>> Technical Manager
> > >>> Qihoo 360 Inc, Shanghai, China
> > >>>
> >
> >
> >
> > --
> > Yizhi Liu
> > DMLC member
> > Technical Manager
> > Qihoo 360 Inc, Shanghai, China
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message