mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Carin Meier <carinme...@gmail.com>
Subject Re: Clojure Package
Date Sun, 10 Jun 2018 12:55:52 GMT
Naveen,

Thanks. I appreciate your help. I acknowledge there are some challenges in
trying to bring in a new language package, but I think if we can make it
work, it will be the best way for it to grow along with the MXNet project
and be a good fit with its goals and ecosystem long term.

I added a confluence page of the general state of what is working and needs
help.
https://cwiki.apache.org/confluence/display/MXNET/Clojure+Package+Contribution+Needs.
Feel free to edit.

Chen,

I added the code generator use on the page as well. I don't fully
understand what you are proposing, but I would like to. If you could
provide some more details to help me, that would be great.

- Carin


On Sun, Jun 10, 2018 at 8:00 AM, Chen HY <chenhy12345@gmail.com> wrote:

> I personally think that is the reason the clojure package should use more
> code generator.
> It makes easier to build everything align with scala package.
>
> 2018-06-10 12:45 GMT+01:00 Naveen Swamy <mnnaveen@gmail.com>:
>
> > I have called for contribution within my organization to help with the
> > review, some of them have shown interest and hope to see them help with
> the
> > review. Carin, I learnt you are popular to have given great talks on
> > Clojure.
> >
> > Could I please ask you document what functionality is working and what
> > parts of it need contribution.
> >
> > In general I want to request a few things:
> > Good API documentation,
> > Examples, turn examples into integration tests.
> > Tests, more tests
> > Everything testable on CI.
> >
> > In the long term, We have to think how to maintain API compatibility and
> > tests across language bindings.
> >
> > Thanks, Naveen
> >
> >
> > On Sat, Jun 9, 2018 at 2:38 AM, YiZhi Liu <eazhi.liu@gmail.com> wrote:
> >
> > > Carin,
> > >
> > > Thank you for the great work. I'll do the review. As I have no
> > > expertise in Clojure, it will really help to have people from Clojure
> > > community reviewing it as well :)
> > >
> > > 2018-06-08 14:22 GMT-07:00 Carin Meier <carinmeier@gmail.com>:
> > > > A couple of questions came up in regard to the PR and the current
> test
> > > > suite state as well as the best way to review the PR since it is a
> new
> > > > language binding.
> > > >
> > > > In regards to the Clojure test suite, most of the Scala test suite
> has
> > > been
> > > > ported over with the goal of having comparable coverage. I can go
> > ahead a
> > > > put in a coverage tool to make that it a bit more transparent.
> > > >
> > > > For reviewing, I have a couple people in the Clojure community that
> are
> > > > interested in collaborating in this project and I can ask them for
> help
> > > in
> > > > reviewing the PR in some way if that is helpful too.
> > > >
> > > > I'm also open to other suggestions.
> > > >
> > > > Thanks,
> > > > Carin
> > > >
> > > > On Fri, Jun 8, 2018 at 4:06 PM, Carin Meier <carinmeier@gmail.com>
> > > wrote:
> > > >
> > > >> Here is the PR https://github.com/apache/incubator-mxnet/pull/11205
> > > >>
> > > >> I've ported in the content from the external github repo (
> > > >> https://github.com/gigasquid/clojure-mxnet), that has been tested
> by
> > > some
> > > >> of the Clojure community, into the contrib directory.
> > > >>
> > > >> There is still lots more to do in relation to adding tests,
> > benchmarks,
> > > >> and increasing stability, but I thought this might be a good point
> to
> > > bring
> > > >> it in initially so that the other work can be reviewed in smaller
> > > chunks.
> > > >> I also would like to get other people involved in making it better,
> > so I
> > > >> thought that having the base package in there would be a good
> starting
> > > >> point for collaboration.
> > > >>
> > > >> Feedback welcome.
> > > >>
> > > >> - Carin
> > > >>
> > > >>
> > > >> On Tue, Jun 5, 2018 at 11:40 AM, Carin Meier <carinmeier@gmail.com>
> > > wrote:
> > > >>
> > > >>> Thanks everyone. I'll work on getting together a PR with your
> > feedback
> > > >>> and post it here.
> > > >>>
> > > >>> On Tue, Jun 5, 2018 at 4:05 AM, Chen HY <chenhy12345@gmail.com>
> > wrote:
> > > >>>
> > > >>>> I would suggest using code generators in case upstream library
> > adding
> > > the
> > > >>>> functions for arrays.
> > > >>>> It seems that cpp binding is using a code generator and works
> fine.
> > > >>>>
> > > >>>> 2018-06-05 7:59 GMT+01:00 Naveen Swamy <mnnaveen@gmail.com>:
> > > >>>>
> > > >>>> > ~/mxnet/contrib/clojure-package good place for the code.
> > > >>>> >
> > > >>>> > the package name org.apache.mxnet.contrib.clojure ? do
you need
> > > mxnet
> > > >>>> > again?
> > > >>>> >
> > > >>>> > I forgot to request to run some benchmarks and document.
One of
> > the
> > > >>>> reasons
> > > >>>> > users use MXNet is because of its performance and we
want to
> > ensure
> > > >>>> that we
> > > >>>> > maintain it across language bindings.
> > > >>>> >
> > > >>>> > Also invite your other clojure programmer buddies to
the party
> :)
> > > >>>> >
> > > >>>> > Thanks, Naveen
> > > >>>> >
> > > >>>> >
> > > >>>> > On Mon, Jun 4, 2018 at 1:55 PM, Carin Meier <
> carinmeier@gmail.com
> > >
> > > >>>> wrote:
> > > >>>> >
> > > >>>> > > Oh right. That's not a problem, I wonder if something
like
> > > >>>> > >
> > > >>>> > > org.apache.mxnet.contrib/clojure-mxnet
> > > >>>> > >
> > > >>>> > > would work?
> > > >>>> > >
> > > >>>> > > If this seems like it is the right direction, we
could work
> out
> > > the
> > > >>>> > details
> > > >>>> > > in a PR.
> > > >>>> > >
> > > >>>> > >
> > > >>>> > > On Mon, Jun 4, 2018 at 4:44 PM, Naveen Swamy <
> > mnnaveen@gmail.com>
> > > >>>> wrote:
> > > >>>> > >
> > > >>>> > > > I agree with your assessment that we shouldn't
need the user
> > to
> > > >>>> change
> > > >>>> > > > their code. I am not sure if we can release
under
> > > >>>> > > org.apache.clojure-mxnet
> > > >>>> > > > we might have to stick with our primary group
id
> > > org.apache.mxnet
> > > >>>> and
> > > >>>> > may
> > > >>>> > > > be create a sub-package under it? any creative
ideas?
> > > >>>> > > >
> > > >>>> > > > On Mon, Jun 4, 2018 at 1:29 PM, Carin Meier
<
> > > carinmeier@gmail.com>
> > > >>>> > > wrote:
> > > >>>> > > >
> > > >>>> > > > > Thanks for the feedback everyone.
> > > >>>> > > > >
> > > >>>> > > > > I agree on the contrib option. I think
it's a great path
> > > forward
> > > >>>> and
> > > >>>> > > > would
> > > >>>> > > > > allow it time for feedback, contribution
by others, and
> > > >>>> > stabilization.
> > > >>>> > > > >
> > > >>>> > > > > If I'm understanding correctly, that would
mean putting
> the
> > > >>>> source
> > > >>>> > code
> > > >>>> > > > in:
> > > >>>> > > > > ~/mxnet/contrib/clojure-package
> > > >>>> > > > >
> > > >>>> > > > > and having the artifact jar named
> > > >>>> > > > > `org.apache.contrib.clojure-mxnet/clojure-mxnet`
> > > >>>> > > > >
> > > >>>> > > > > I would recommend not having the individual
namespaces of
> > the
> > > >>>> files
> > > >>>> > > have
> > > >>>> > > > > contrib embedded in them, so that if it
graduates, users
> > won't
> > > >>>> have
> > > >>>> > to
> > > >>>> > > > > change their code, only the dependency.
> > > >>>> > > > >
> > > >>>> > > > > Please let me know if this is correct
or if there are any
> > > other
> > > >>>> > ideas.
> > > >>>> > > > >
> > > >>>> > > > > - Carin
> > > >>>> > > > >
> > > >>>> > > > >
> > > >>>> > > > >
> > > >>>> > > > >
> > > >>>> > > > > On Mon, Jun 4, 2018 at 4:03 PM, Naveen
Swamy <
> > > mnnaveen@gmail.com
> > > >>>> >
> > > >>>> > > wrote:
> > > >>>> > > > >
> > > >>>> > > > > > I think that's a great idea to bring
in under contrib
> and
> > we
> > > >>>> can
> > > >>>> > also
> > > >>>> > > > get
> > > >>>> > > > > > user feedback
> > > >>>> > > > > >
> > > >>>> > > > > > > On Jun 4, 2018, at 12:44 PM,
sandeep krishnamurthy <
> > > >>>> > > > > > sandeep.krishna98@gmail.com> wrote:
> > > >>>> > > > > > >
> > > >>>> > > > > > > Hi Carin,
> > > >>>> > > > > > >
> > > >>>> > > > > > > This is a commendable work.
Thanks a lot for all the
> > hard
> > > and
> > > >>>> > smart
> > > >>>> > > > > work
> > > >>>> > > > > > > you have put behind this :-)
I think this will be a
> > great
> > > >>>> value
> > > >>>> > > > > addition.
> > > >>>> > > > > > >
> > > >>>> > > > > > > If people are not sure about
usage, can I suggest this
> > > >>>> awesome
> > > >>>> > work
> > > >>>> > > > to
> > > >>>> > > > > be
> > > >>>> > > > > > > brought in "contrib" package?
Invite and build the
> > > community
> > > >>>> > around
> > > >>>> > > > > > > Clojure, stabilize and increase
the coverage, and
> based
> > on
> > > >>>> usage
> > > >>>> > > and
> > > >>>> > > > > > > development, graduate it to
main stable support from
> > > contrib.
> > > >>>> > > > > > >
> > > >>>> > > > > > > Suggestions and thoughts?
> > > >>>> > > > > > >
> > > >>>> > > > > > > Best,
> > > >>>> > > > > > > Sandeep
> > > >>>> > > > > > >
> > > >>>> > > > > > >
> > > >>>> > > > > > >
> > > >>>> > > > > > > On Mon, Jun 4, 2018 at 12:27
PM, Ivan Serdyuk <
> > > >>>> > > > > > local.tourist.kiev@gmail.com>
> > > >>>> > > > > > > wrote:
> > > >>>> > > > > > >
> > > >>>> > > > > > >> Hello everyone.
> > > >>>> > > > > > >>
> > > >>>> > > > > > >> A small comment, about Scala
API: main commiters are
> > > hardly
> > > >>>> > > > available,
> > > >>>> > > > > > as
> > > >>>> > > > > > >> for today.
> > > >>>> > > > > > >>
> > > >>>> > > > > > >> As for Clojure - I might
suggest that it might be
> > > possible
> > > >>>> to
> > > >>>> > > > enlight
> > > >>>> > > > > > >> future work, for that package,
for Clojure
> developers.
> > > >>>> > > > > > >>
> > > >>>> > > > > > >>> On Mon, Jun 4, 2018
at 8:50 PM, Naveen Swamy <
> > > >>>> > mnnaveen@gmail.com
> > > >>>> > > >
> > > >>>> > > > > > wrote:
> > > >>>> > > > > > >>>
> > > >>>> > > > > > >>> Hi Carin,
> > > >>>> > > > > > >>>
> > > >>>> > > > > > >>> This is great effort
and very nicely documented.
> When
> > I
> > > >>>> > surveyed
> > > >>>> > > at
> > > >>>> > > > > my
> > > >>>> > > > > > >> day
> > > >>>> > > > > > >>> job on the interest
for Scala packages there were a
> > few
> > > >>>> people
> > > >>>> > > who
> > > >>>> > > > > > >>> specifically were interested
in the Clojure
> packages,
> > I
> > > >>>> think
> > > >>>> > > those
> > > >>>> > > > > are
> > > >>>> > > > > > >>> people who might have
tried and understand the
> > > complexity
> > > >>>> of
> > > >>>> > > making
> > > >>>> > > > > > Scala
> > > >>>> > > > > > >>> work in Clojure.
> > > >>>> > > > > > >>>
> > > >>>> > > > > > >>> I am in support of a
MXNet-Clojure package, I
> request
> > > you
> > > >>>> to
> > > >>>> > add
> > > >>>> > > > more
> > > >>>> > > > > > >> unit
> > > >>>> > > > > > >>> tests and integration
tests that can be ported to
> CI.
> > > >>>> > > > > > >>>
> > > >>>> > > > > > >>> Thanks for your efforts.
> > > >>>> > > > > > >>>
> > > >>>> > > > > > >>> -Naveen
> > > >>>> > > > > > >>>
> > > >>>> > > > > > >>>
> > > >>>> > > > > > >>>
> > > >>>> > > > > > >>> On Fri, Jun 1, 2018
at 6:16 PM, Carin Meier <
> > > >>>> > > carinmeier@gmail.com>
> > > >>>> > > > > > >> wrote:
> > > >>>> > > > > > >>>
> > > >>>> > > > > > >>>> It is always a good
thing to consider the cost with
> > the
> > > >>>> > benefit.
> > > >>>> > > > > I'll
> > > >>>> > > > > > >> do
> > > >>>> > > > > > >>> my
> > > >>>> > > > > > >>>> best to explain
what I see the tradeoffs to be.
> > > >>>> > > > > > >>>>
> > > >>>> > > > > > >>>> First, I wanted
to clarify that it took significant
> > > >>>> > development
> > > >>>> > > > > effort
> > > >>>> > > > > > >> to
> > > >>>> > > > > > >>>> get the Clojure
package and the interop working
> > > properly
> > > >>>> > despite
> > > >>>> > > > my
> > > >>>> > > > > > >>> simple
> > > >>>> > > > > > >>>> looking design on
the confluence page :)
> > > >>>> > > > > > >>>>
> > > >>>> > > > > > >>>> One of the advantages
of MXNet over its competitors
> > is
> > > its
> > > >>>> > many
> > > >>>> > > > > > >> language
> > > >>>> > > > > > >>>> support. The Clojure
package would only increase
> the
> > > >>>> value of
> > > >>>> > > this
> > > >>>> > > > > > >>>> proposition and
bring new users and growth into the
> > > >>>> community.
> > > >>>> > > > > > >>>> However, there is
a cost associated with adding
> this
> > > >>>> language
> > > >>>> > > > > support
> > > >>>> > > > > > >> as
> > > >>>> > > > > > >>>> you pointed out.
> > > >>>> > > > > > >>>>
> > > >>>> > > > > > >>>> Since the Clojure
package right now is only reliant
> > on
> > > the
> > > >>>> > Scala
> > > >>>> > > > > jars
> > > >>>> > > > > > >>> from
> > > >>>> > > > > > >>>> Maven, it can exist
outside the main project as an
> > > >>>> independent
> > > >>>> > > > repo
> > > >>>> > > > > > >> but I
> > > >>>> > > > > > >>>> think that would
lessen the growth benefit both to
> > the
> > > >>>> Clojure
> > > >>>> > > > > > >> community
> > > >>>> > > > > > >>>> and to the MXNet
community to not be included as a
> > > first
> > > >>>> class
> > > >>>> > > > > > >> language.
> > > >>>> > > > > > >>>>
> > > >>>> > > > > > >>>> I believe having
first class Clojure support in
> MXNet
> > > is
> > > >>>> > > valuable,
> > > >>>> > > > > but
> > > >>>> > > > > > >>> the
> > > >>>> > > > > > >>>> cost of that support
is up to the community to
> > decide.
> > > >>>> > > > > > >>>>
> > > >>>> > > > > > >>>> Is there a process
for considering a new package in
> > > MXNet?
> > > >>>> > > > > > >>>>
> > > >>>> > > > > > >>>> - Carin
> > > >>>> > > > > > >>>>
> > > >>>> > > > > > >>>>> On Fri, Jun
1, 2018 at 5:51 PM, Chen HY <
> > > >>>> > chenhy12345@gmail.com
> > > >>>> > > >
> > > >>>> > > > > > wrote:
> > > >>>> > > > > > >>>>>
> > > >>>> > > > > > >>>>> Have checked
the issue and the confluence page,
> but
> > > still
> > > >>>> > > > curious.
> > > >>>> > > > > > >>>>> Clojure and
Scala are both JVM based languages.
> > > >>>> > > > > > >>>>> They, as well
as many JVM based languages, can
> share
> > > >>>> their
> > > >>>> > > class
> > > >>>> > > > > and
> > > >>>> > > > > > >>>> method
> > > >>>> > > > > > >>>>> at a certain
level.
> > > >>>> > > > > > >>>>> Why should the
community maintain two APIs for two
> > > >>>> languages
> > > >>>> > > with
> > > >>>> > > > > can
> > > >>>> > > > > > >>>> share
> > > >>>> > > > > > >>>>> their packages
with almost zero effort?
> > > >>>> > > > > > >>>>>
> > > >>>> > > > > > >>>>>
> > > >>>> > > > > > >>>>> 2018-06-01 21:58
GMT+01:00 Carin Meier <
> > > >>>> carinmeier@gmail.com
> > > >>>> > >:
> > > >>>> > > > > > >>>>>
> > > >>>> > > > > > >>>>>> Hi all,
> > > >>>> > > > > > >>>>>>
> > > >>>> > > > > > >>>>>> I've been
working on a Clojure package for MXNet.
> > > Since
> > > >>>> > > Clojure
> > > >>>> > > > is
> > > >>>> > > > > > >> a
> > > >>>> > > > > > >>>> JVM
> > > >>>> > > > > > >>>>>> language,
the package leverages the great work of
> > the
> > > >>>> > existing
> > > >>>> > > > > > >> Scala
> > > >>>> > > > > > >>>>>> package.
> > > >>>> > > > > > >>>>>>
> > > >>>> > > > > > >>>>>> I would
appreciate any feedback and testing.
> > > >>>> > > > > > >>>>>>
> > > >>>> > > > > > >>>>>> Here is
the original issue:
> > > >>>> > > > > > >>>>>> https://github.com/apache/
> > > incubator-mxnet/issues/8971
> > > >>>> > > > > > >>>>>>
> > > >>>> > > > > > >>>>>> Architecture
& Design:
> > > >>>> > > > > > >>>>>> https://cwiki.apache.org/
> confluence/display/MXNET/
> > > >>>> > > MXNet+Clojure
> > > >>>> > > > > > >>>>>>
> > > >>>> > > > > > >>>>>> and the
github repo for rapid testing and issue
> > > fixing
> > > >>>> > before
> > > >>>> > > of
> > > >>>> > > > > > >>>> opening
> > > >>>> > > > > > >>>>> an
> > > >>>> > > > > > >>>>>> official
PR https://github.com/gigasquid/
> > > clojure-mxnet
> > > >>>> > > > > > >>>>>>
> > > >>>> > > > > > >>>>>> I'm also
active in the slack channel so feel free
> > to
> > > >>>> ping me
> > > >>>> > > > > there.
> > > >>>> > > > > > >>>>>>
> > > >>>> > > > > > >>>>>> Thanks,
> > > >>>> > > > > > >>>>>> Carin Meier
> > > >>>> > > > > > >>>>>>
> > > >>>> > > > > > >>>>>
> > > >>>> > > > > > >>>>>
> > > >>>> > > > > > >>>>>
> > > >>>> > > > > > >>>>> --
> > > >>>> > > > > > >>>>> Chen Hanyang
陈涵洋
> > > >>>> > > > > > >>>>> Software School
Fudan University
> > > >>>> > > > > > >>>>> +86-138-1881-7745
> > > >>>> > > > > > >>>>>
> > > >>>> > > > > > >>>>
> > > >>>> > > > > > >>>
> > > >>>> > > > > > >>
> > > >>>> > > > > > >
> > > >>>> > > > > > >
> > > >>>> > > > > > >
> > > >>>> > > > > > > --
> > > >>>> > > > > > > Sandeep Krishnamurthy
> > > >>>> > > > > >
> > > >>>> > > > >
> > > >>>> > > >
> > > >>>> > >
> > > >>>> >
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> Chen Hanyang 陈涵洋
> > > >>>> Software School Fudan University
> > > >>>> +86-138-1881-7745
> > > >>>>
> > > >>>
> > > >>>
> > > >>
> > >
> > >
> > >
> > > --
> > > Yizhi Liu
> > > DMLC member
> > > Amazon Web Services
> > > Vancouver, Canada
> > >
> >
>
>
>
> --
> Chen Hanyang 陈涵洋
> Software School Fudan University
> +86-138-1881-7745
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message