mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen HY <chenhy12...@gmail.com>
Subject Re: Clojure Package
Date Sun, 10 Jun 2018 13:40:05 GMT
Yep. An example is the cpp port used following code to generate op.h
automatically.

https://github.com/apache/incubator-mxnet/blob/master/cpp-package/scripts/OpWrapperGenerator.py

It makes easier for everyone to follow up with the latest APIs.
It is disappointing if the python version supports the latest API but the
version of your favorite language doesn't.

2018-06-10 14:19 GMT+01:00 Naveen Swamy <mnnaveen@gmail.com>:

> this is fantastic, thanks for being a champ. I am glad to have you join us
> :). Yes, we would love to make it work and be available for our users, you
> can make features available iteratively. I understand you alone can't have
> everything in one go, I am hoping more contributors will join the effort.
>
> My guess is Chen is recommending generation of Clojure code from Scala.
>
> On Sun, Jun 10, 2018 at 2:55 PM, Carin Meier <carinmeier@gmail.com> wrote:
>
> > Naveen,
> >
> > Thanks. I appreciate your help. I acknowledge there are some challenges
> in
> > trying to bring in a new language package, but I think if we can make it
> > work, it will be the best way for it to grow along with the MXNet project
> > and be a good fit with its goals and ecosystem long term.
> >
> > I added a confluence page of the general state of what is working and
> needs
> > help.
> > https://cwiki.apache.org/confluence/display/MXNET/
> > Clojure+Package+Contribution+Needs.
> > Feel free to edit.
> >
> > Chen,
> >
> > I added the code generator use on the page as well. I don't fully
> > understand what you are proposing, but I would like to. If you could
> > provide some more details to help me, that would be great.
> >
> > - Carin
> >
> >
> > On Sun, Jun 10, 2018 at 8:00 AM, Chen HY <chenhy12345@gmail.com> wrote:
> >
> > > I personally think that is the reason the clojure package should use
> more
> > > code generator.
> > > It makes easier to build everything align with scala package.
> > >
> > > 2018-06-10 12:45 GMT+01:00 Naveen Swamy <mnnaveen@gmail.com>:
> > >
> > > > I have called for contribution within my organization to help with
> the
> > > > review, some of them have shown interest and hope to see them help
> with
> > > the
> > > > review. Carin, I learnt you are popular to have given great talks on
> > > > Clojure.
> > > >
> > > > Could I please ask you document what functionality is working and
> what
> > > > parts of it need contribution.
> > > >
> > > > In general I want to request a few things:
> > > > Good API documentation,
> > > > Examples, turn examples into integration tests.
> > > > Tests, more tests
> > > > Everything testable on CI.
> > > >
> > > > In the long term, We have to think how to maintain API compatibility
> > and
> > > > tests across language bindings.
> > > >
> > > > Thanks, Naveen
> > > >
> > > >
> > > > On Sat, Jun 9, 2018 at 2:38 AM, YiZhi Liu <eazhi.liu@gmail.com>
> wrote:
> > > >
> > > > > Carin,
> > > > >
> > > > > Thank you for the great work. I'll do the review. As I have no
> > > > > expertise in Clojure, it will really help to have people from
> Clojure
> > > > > community reviewing it as well :)
> > > > >
> > > > > 2018-06-08 14:22 GMT-07:00 Carin Meier <carinmeier@gmail.com>:
> > > > > > A couple of questions came up in regard to the PR and the current
> > > test
> > > > > > suite state as well as the best way to review the PR since it
is
> a
> > > new
> > > > > > language binding.
> > > > > >
> > > > > > In regards to the Clojure test suite, most of the Scala test
> suite
> > > has
> > > > > been
> > > > > > ported over with the goal of having comparable coverage. I can
go
> > > > ahead a
> > > > > > put in a coverage tool to make that it a bit more transparent.
> > > > > >
> > > > > > For reviewing, I have a couple people in the Clojure community
> that
> > > are
> > > > > > interested in collaborating in this project and I can ask them
> for
> > > help
> > > > > in
> > > > > > reviewing the PR in some way if that is helpful too.
> > > > > >
> > > > > > I'm also open to other suggestions.
> > > > > >
> > > > > > Thanks,
> > > > > > Carin
> > > > > >
> > > > > > On Fri, Jun 8, 2018 at 4:06 PM, Carin Meier <
> carinmeier@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > >> Here is the PR https://github.com/apache/
> > incubator-mxnet/pull/11205
> > > > > >>
> > > > > >> I've ported in the content from the external github repo
(
> > > > > >> https://github.com/gigasquid/clojure-mxnet), that has been
> tested
> > > by
> > > > > some
> > > > > >> of the Clojure community, into the contrib directory.
> > > > > >>
> > > > > >> There is still lots more to do in relation to adding tests,
> > > > benchmarks,
> > > > > >> and increasing stability, but I thought this might be a
good
> point
> > > to
> > > > > bring
> > > > > >> it in initially so that the other work can be reviewed in
> smaller
> > > > > chunks.
> > > > > >> I also would like to get other people involved in making
it
> > better,
> > > > so I
> > > > > >> thought that having the base package in there would be a
good
> > > starting
> > > > > >> point for collaboration.
> > > > > >>
> > > > > >> Feedback welcome.
> > > > > >>
> > > > > >> - Carin
> > > > > >>
> > > > > >>
> > > > > >> On Tue, Jun 5, 2018 at 11:40 AM, Carin Meier <
> > carinmeier@gmail.com>
> > > > > wrote:
> > > > > >>
> > > > > >>> Thanks everyone. I'll work on getting together a PR
with your
> > > > feedback
> > > > > >>> and post it here.
> > > > > >>>
> > > > > >>> On Tue, Jun 5, 2018 at 4:05 AM, Chen HY <chenhy12345@gmail.com
> >
> > > > wrote:
> > > > > >>>
> > > > > >>>> I would suggest using code generators in case upstream
library
> > > > adding
> > > > > the
> > > > > >>>> functions for arrays.
> > > > > >>>> It seems that cpp binding is using a code generator
and works
> > > fine.
> > > > > >>>>
> > > > > >>>> 2018-06-05 7:59 GMT+01:00 Naveen Swamy <mnnaveen@gmail.com>:
> > > > > >>>>
> > > > > >>>> > ~/mxnet/contrib/clojure-package good place
for the code.
> > > > > >>>> >
> > > > > >>>> > the package name org.apache.mxnet.contrib.clojure
? do you
> > need
> > > > > mxnet
> > > > > >>>> > again?
> > > > > >>>> >
> > > > > >>>> > I forgot to request to run some benchmarks
and document. One
> > of
> > > > the
> > > > > >>>> reasons
> > > > > >>>> > users use MXNet is because of its performance
and we want to
> > > > ensure
> > > > > >>>> that we
> > > > > >>>> > maintain it across language bindings.
> > > > > >>>> >
> > > > > >>>> > Also invite your other clojure programmer buddies
to the
> party
> > > :)
> > > > > >>>> >
> > > > > >>>> > Thanks, Naveen
> > > > > >>>> >
> > > > > >>>> >
> > > > > >>>> > On Mon, Jun 4, 2018 at 1:55 PM, Carin Meier
<
> > > carinmeier@gmail.com
> > > > >
> > > > > >>>> wrote:
> > > > > >>>> >
> > > > > >>>> > > Oh right. That's not a problem, I wonder
if something like
> > > > > >>>> > >
> > > > > >>>> > > org.apache.mxnet.contrib/clojure-mxnet
> > > > > >>>> > >
> > > > > >>>> > > would work?
> > > > > >>>> > >
> > > > > >>>> > > If this seems like it is the right direction,
we could
> work
> > > out
> > > > > the
> > > > > >>>> > details
> > > > > >>>> > > in a PR.
> > > > > >>>> > >
> > > > > >>>> > >
> > > > > >>>> > > On Mon, Jun 4, 2018 at 4:44 PM, Naveen
Swamy <
> > > > mnnaveen@gmail.com>
> > > > > >>>> wrote:
> > > > > >>>> > >
> > > > > >>>> > > > I agree with your assessment that
we shouldn't need the
> > user
> > > > to
> > > > > >>>> change
> > > > > >>>> > > > their code. I am not sure if we can
release under
> > > > > >>>> > > org.apache.clojure-mxnet
> > > > > >>>> > > > we might have to stick with our primary
group id
> > > > > org.apache.mxnet
> > > > > >>>> and
> > > > > >>>> > may
> > > > > >>>> > > > be create a sub-package under it?
any creative ideas?
> > > > > >>>> > > >
> > > > > >>>> > > > On Mon, Jun 4, 2018 at 1:29 PM, Carin
Meier <
> > > > > carinmeier@gmail.com>
> > > > > >>>> > > wrote:
> > > > > >>>> > > >
> > > > > >>>> > > > > Thanks for the feedback everyone.
> > > > > >>>> > > > >
> > > > > >>>> > > > > I agree on the contrib option.
I think it's a great
> path
> > > > > forward
> > > > > >>>> and
> > > > > >>>> > > > would
> > > > > >>>> > > > > allow it time for feedback,
contribution by others,
> and
> > > > > >>>> > stabilization.
> > > > > >>>> > > > >
> > > > > >>>> > > > > If I'm understanding correctly,
that would mean
> putting
> > > the
> > > > > >>>> source
> > > > > >>>> > code
> > > > > >>>> > > > in:
> > > > > >>>> > > > > ~/mxnet/contrib/clojure-package
> > > > > >>>> > > > >
> > > > > >>>> > > > > and having the artifact jar
named
> > > > > >>>> > > > > `org.apache.contrib.clojure-mxnet/clojure-mxnet`
> > > > > >>>> > > > >
> > > > > >>>> > > > > I would recommend not having
the individual namespaces
> > of
> > > > the
> > > > > >>>> files
> > > > > >>>> > > have
> > > > > >>>> > > > > contrib embedded in them, so
that if it graduates,
> users
> > > > won't
> > > > > >>>> have
> > > > > >>>> > to
> > > > > >>>> > > > > change their code, only the
dependency.
> > > > > >>>> > > > >
> > > > > >>>> > > > > Please let me know if this is
correct or if there are
> > any
> > > > > other
> > > > > >>>> > ideas.
> > > > > >>>> > > > >
> > > > > >>>> > > > > - Carin
> > > > > >>>> > > > >
> > > > > >>>> > > > >
> > > > > >>>> > > > >
> > > > > >>>> > > > >
> > > > > >>>> > > > > On Mon, Jun 4, 2018 at 4:03
PM, Naveen Swamy <
> > > > > mnnaveen@gmail.com
> > > > > >>>> >
> > > > > >>>> > > wrote:
> > > > > >>>> > > > >
> > > > > >>>> > > > > > I think that's a great
idea to bring in under
> contrib
> > > and
> > > > we
> > > > > >>>> can
> > > > > >>>> > also
> > > > > >>>> > > > get
> > > > > >>>> > > > > > user feedback
> > > > > >>>> > > > > >
> > > > > >>>> > > > > > > On Jun 4, 2018, at
12:44 PM, sandeep
> krishnamurthy <
> > > > > >>>> > > > > > sandeep.krishna98@gmail.com>
wrote:
> > > > > >>>> > > > > > >
> > > > > >>>> > > > > > > Hi Carin,
> > > > > >>>> > > > > > >
> > > > > >>>> > > > > > > This is a commendable
work. Thanks a lot for all
> the
> > > > hard
> > > > > and
> > > > > >>>> > smart
> > > > > >>>> > > > > work
> > > > > >>>> > > > > > > you have put behind
this :-) I think this will be
> a
> > > > great
> > > > > >>>> value
> > > > > >>>> > > > > addition.
> > > > > >>>> > > > > > >
> > > > > >>>> > > > > > > If people are not
sure about usage, can I suggest
> > this
> > > > > >>>> awesome
> > > > > >>>> > work
> > > > > >>>> > > > to
> > > > > >>>> > > > > be
> > > > > >>>> > > > > > > brought in "contrib"
package? Invite and build the
> > > > > community
> > > > > >>>> > around
> > > > > >>>> > > > > > > Clojure, stabilize
and increase the coverage, and
> > > based
> > > > on
> > > > > >>>> usage
> > > > > >>>> > > and
> > > > > >>>> > > > > > > development, graduate
it to main stable support
> from
> > > > > contrib.
> > > > > >>>> > > > > > >
> > > > > >>>> > > > > > > Suggestions and thoughts?
> > > > > >>>> > > > > > >
> > > > > >>>> > > > > > > Best,
> > > > > >>>> > > > > > > Sandeep
> > > > > >>>> > > > > > >
> > > > > >>>> > > > > > >
> > > > > >>>> > > > > > >
> > > > > >>>> > > > > > > On Mon, Jun 4, 2018
at 12:27 PM, Ivan Serdyuk <
> > > > > >>>> > > > > > local.tourist.kiev@gmail.com>
> > > > > >>>> > > > > > > wrote:
> > > > > >>>> > > > > > >
> > > > > >>>> > > > > > >> Hello everyone.
> > > > > >>>> > > > > > >>
> > > > > >>>> > > > > > >> A small comment,
about Scala API: main commiters
> > are
> > > > > hardly
> > > > > >>>> > > > available,
> > > > > >>>> > > > > > as
> > > > > >>>> > > > > > >> for today.
> > > > > >>>> > > > > > >>
> > > > > >>>> > > > > > >> As for Clojure
- I might suggest that it might be
> > > > > possible
> > > > > >>>> to
> > > > > >>>> > > > enlight
> > > > > >>>> > > > > > >> future work, for
that package, for Clojure
> > > developers.
> > > > > >>>> > > > > > >>
> > > > > >>>> > > > > > >>> On Mon, Jun
4, 2018 at 8:50 PM, Naveen Swamy <
> > > > > >>>> > mnnaveen@gmail.com
> > > > > >>>> > > >
> > > > > >>>> > > > > > wrote:
> > > > > >>>> > > > > > >>>
> > > > > >>>> > > > > > >>> Hi Carin,
> > > > > >>>> > > > > > >>>
> > > > > >>>> > > > > > >>> This is great
effort and very nicely documented.
> > > When
> > > > I
> > > > > >>>> > surveyed
> > > > > >>>> > > at
> > > > > >>>> > > > > my
> > > > > >>>> > > > > > >> day
> > > > > >>>> > > > > > >>> job on the
interest for Scala packages there
> were
> > a
> > > > few
> > > > > >>>> people
> > > > > >>>> > > who
> > > > > >>>> > > > > > >>> specifically
were interested in the Clojure
> > > packages,
> > > > I
> > > > > >>>> think
> > > > > >>>> > > those
> > > > > >>>> > > > > are
> > > > > >>>> > > > > > >>> people who
might have tried and understand the
> > > > > complexity
> > > > > >>>> of
> > > > > >>>> > > making
> > > > > >>>> > > > > > Scala
> > > > > >>>> > > > > > >>> work in Clojure.
> > > > > >>>> > > > > > >>>
> > > > > >>>> > > > > > >>> I am in support
of a MXNet-Clojure package, I
> > > request
> > > > > you
> > > > > >>>> to
> > > > > >>>> > add
> > > > > >>>> > > > more
> > > > > >>>> > > > > > >> unit
> > > > > >>>> > > > > > >>> tests and
integration tests that can be ported
> to
> > > CI.
> > > > > >>>> > > > > > >>>
> > > > > >>>> > > > > > >>> Thanks for
your efforts.
> > > > > >>>> > > > > > >>>
> > > > > >>>> > > > > > >>> -Naveen
> > > > > >>>> > > > > > >>>
> > > > > >>>> > > > > > >>>
> > > > > >>>> > > > > > >>>
> > > > > >>>> > > > > > >>> On Fri, Jun
1, 2018 at 6:16 PM, Carin Meier <
> > > > > >>>> > > carinmeier@gmail.com>
> > > > > >>>> > > > > > >> wrote:
> > > > > >>>> > > > > > >>>
> > > > > >>>> > > > > > >>>> It is
always a good thing to consider the cost
> > with
> > > > the
> > > > > >>>> > benefit.
> > > > > >>>> > > > > I'll
> > > > > >>>> > > > > > >> do
> > > > > >>>> > > > > > >>> my
> > > > > >>>> > > > > > >>>> best to
explain what I see the tradeoffs to be.
> > > > > >>>> > > > > > >>>>
> > > > > >>>> > > > > > >>>> First,
I wanted to clarify that it took
> > significant
> > > > > >>>> > development
> > > > > >>>> > > > > effort
> > > > > >>>> > > > > > >> to
> > > > > >>>> > > > > > >>>> get the
Clojure package and the interop working
> > > > > properly
> > > > > >>>> > despite
> > > > > >>>> > > > my
> > > > > >>>> > > > > > >>> simple
> > > > > >>>> > > > > > >>>> looking
design on the confluence page :)
> > > > > >>>> > > > > > >>>>
> > > > > >>>> > > > > > >>>> One of
the advantages of MXNet over its
> > competitors
> > > > is
> > > > > its
> > > > > >>>> > many
> > > > > >>>> > > > > > >> language
> > > > > >>>> > > > > > >>>> support.
The Clojure package would only
> increase
> > > the
> > > > > >>>> value of
> > > > > >>>> > > this
> > > > > >>>> > > > > > >>>> proposition
and bring new users and growth into
> > the
> > > > > >>>> community.
> > > > > >>>> > > > > > >>>> However,
there is a cost associated with adding
> > > this
> > > > > >>>> language
> > > > > >>>> > > > > support
> > > > > >>>> > > > > > >> as
> > > > > >>>> > > > > > >>>> you pointed
out.
> > > > > >>>> > > > > > >>>>
> > > > > >>>> > > > > > >>>> Since
the Clojure package right now is only
> > reliant
> > > > on
> > > > > the
> > > > > >>>> > Scala
> > > > > >>>> > > > > jars
> > > > > >>>> > > > > > >>> from
> > > > > >>>> > > > > > >>>> Maven,
it can exist outside the main project as
> > an
> > > > > >>>> independent
> > > > > >>>> > > > repo
> > > > > >>>> > > > > > >> but I
> > > > > >>>> > > > > > >>>> think
that would lessen the growth benefit both
> > to
> > > > the
> > > > > >>>> Clojure
> > > > > >>>> > > > > > >> community
> > > > > >>>> > > > > > >>>> and to
the MXNet community to not be included
> as
> > a
> > > > > first
> > > > > >>>> class
> > > > > >>>> > > > > > >> language.
> > > > > >>>> > > > > > >>>>
> > > > > >>>> > > > > > >>>> I believe
having first class Clojure support in
> > > MXNet
> > > > > is
> > > > > >>>> > > valuable,
> > > > > >>>> > > > > but
> > > > > >>>> > > > > > >>> the
> > > > > >>>> > > > > > >>>> cost of
that support is up to the community to
> > > > decide.
> > > > > >>>> > > > > > >>>>
> > > > > >>>> > > > > > >>>> Is there
a process for considering a new
> package
> > in
> > > > > MXNet?
> > > > > >>>> > > > > > >>>>
> > > > > >>>> > > > > > >>>> - Carin
> > > > > >>>> > > > > > >>>>
> > > > > >>>> > > > > > >>>>> On
Fri, Jun 1, 2018 at 5:51 PM, Chen HY <
> > > > > >>>> > chenhy12345@gmail.com
> > > > > >>>> > > >
> > > > > >>>> > > > > > wrote:
> > > > > >>>> > > > > > >>>>>
> > > > > >>>> > > > > > >>>>> Have
checked the issue and the confluence
> page,
> > > but
> > > > > still
> > > > > >>>> > > > curious.
> > > > > >>>> > > > > > >>>>> Clojure
and Scala are both JVM based
> languages.
> > > > > >>>> > > > > > >>>>> They,
as well as many JVM based languages, can
> > > share
> > > > > >>>> their
> > > > > >>>> > > class
> > > > > >>>> > > > > and
> > > > > >>>> > > > > > >>>> method
> > > > > >>>> > > > > > >>>>> at
a certain level.
> > > > > >>>> > > > > > >>>>> Why
should the community maintain two APIs for
> > two
> > > > > >>>> languages
> > > > > >>>> > > with
> > > > > >>>> > > > > can
> > > > > >>>> > > > > > >>>> share
> > > > > >>>> > > > > > >>>>> their
packages with almost zero effort?
> > > > > >>>> > > > > > >>>>>
> > > > > >>>> > > > > > >>>>>
> > > > > >>>> > > > > > >>>>> 2018-06-01
21:58 GMT+01:00 Carin Meier <
> > > > > >>>> carinmeier@gmail.com
> > > > > >>>> > >:
> > > > > >>>> > > > > > >>>>>
> > > > > >>>> > > > > > >>>>>>
Hi all,
> > > > > >>>> > > > > > >>>>>>
> > > > > >>>> > > > > > >>>>>>
I've been working on a Clojure package for
> > MXNet.
> > > > > Since
> > > > > >>>> > > Clojure
> > > > > >>>> > > > is
> > > > > >>>> > > > > > >> a
> > > > > >>>> > > > > > >>>> JVM
> > > > > >>>> > > > > > >>>>>>
language, the package leverages the great
> work
> > of
> > > > the
> > > > > >>>> > existing
> > > > > >>>> > > > > > >> Scala
> > > > > >>>> > > > > > >>>>>>
package.
> > > > > >>>> > > > > > >>>>>>
> > > > > >>>> > > > > > >>>>>>
I would appreciate any feedback and testing.
> > > > > >>>> > > > > > >>>>>>
> > > > > >>>> > > > > > >>>>>>
Here is the original issue:
> > > > > >>>> > > > > > >>>>>>
https://github.com/apache/
> > > > > incubator-mxnet/issues/8971
> > > > > >>>> > > > > > >>>>>>
> > > > > >>>> > > > > > >>>>>>
Architecture & Design:
> > > > > >>>> > > > > > >>>>>>
https://cwiki.apache.org/
> > > confluence/display/MXNET/
> > > > > >>>> > > MXNet+Clojure
> > > > > >>>> > > > > > >>>>>>
> > > > > >>>> > > > > > >>>>>>
and the github repo for rapid testing and
> issue
> > > > > fixing
> > > > > >>>> > before
> > > > > >>>> > > of
> > > > > >>>> > > > > > >>>> opening
> > > > > >>>> > > > > > >>>>> an
> > > > > >>>> > > > > > >>>>>>
official PR https://github.com/gigasquid/
> > > > > clojure-mxnet
> > > > > >>>> > > > > > >>>>>>
> > > > > >>>> > > > > > >>>>>>
I'm also active in the slack channel so feel
> > free
> > > > to
> > > > > >>>> ping me
> > > > > >>>> > > > > there.
> > > > > >>>> > > > > > >>>>>>
> > > > > >>>> > > > > > >>>>>>
Thanks,
> > > > > >>>> > > > > > >>>>>>
Carin Meier
> > > > > >>>> > > > > > >>>>>>
> > > > > >>>> > > > > > >>>>>
> > > > > >>>> > > > > > >>>>>
> > > > > >>>> > > > > > >>>>>
> > > > > >>>> > > > > > >>>>> --
> > > > > >>>> > > > > > >>>>> Chen
Hanyang 陈涵洋
> > > > > >>>> > > > > > >>>>> Software
School Fudan University
> > > > > >>>> > > > > > >>>>> +86-138-1881-7745
> > > > > >>>> > > > > > >>>>>
> > > > > >>>> > > > > > >>>>
> > > > > >>>> > > > > > >>>
> > > > > >>>> > > > > > >>
> > > > > >>>> > > > > > >
> > > > > >>>> > > > > > >
> > > > > >>>> > > > > > >
> > > > > >>>> > > > > > > --
> > > > > >>>> > > > > > > Sandeep Krishnamurthy
> > > > > >>>> > > > > >
> > > > > >>>> > > > >
> > > > > >>>> > > >
> > > > > >>>> > >
> > > > > >>>> >
> > > > > >>>>
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> --
> > > > > >>>> Chen Hanyang 陈涵洋
> > > > > >>>> Software School Fudan University
> > > > > >>>> +86-138-1881-7745
> > > > > >>>>
> > > > > >>>
> > > > > >>>
> > > > > >>
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Yizhi Liu
> > > > > DMLC member
> > > > > Amazon Web Services
> > > > > Vancouver, Canada
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Chen Hanyang 陈涵洋
> > > Software School Fudan University
> > > +86-138-1881-7745
> > >
> >
>



-- 
Chen Hanyang 陈涵洋
Software School Fudan University
+86-138-1881-7745

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message