mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pedro Larroy <pedro.larroy.li...@gmail.com>
Subject Re: [Proposal] New operator graph for MXNet
Date Wed, 15 May 2019 00:50:49 GMT
Hi Tianqi

Thanks for the quick response.

Could you point to examples where graph.h is being exposed which would
not be possible with what I propose? I don't think my proposal is
having any impact in language bindings, and the way I describe it
doesn't affect having or not having higher language bindings. Please
elaborate so I can understand your concern.  Maybe code examples where
the graph attributes are being changed from Python?  I don't think we
have this on MXNet. This is such a core foundation for MXNet, that I
don't think we should compromise on it because other project not
directly related to MXNet might want to expose some untyped graph and
Node attributes.  The current status makes maintaining the code very
painful and also is preventing desired features such as higher order
gradients to be developed. I have heard from you many times how speed
is critical for us to innovate in this quickly changing field.

My proposal is limited to the graph and wouldn't change the way
operators are registered and arguments are processed for operators for
example.


Regarding the second point, the documentation about Relay in the web
which I found for example:

https://docs.tvm.ai/dev/relay_add_op.html#

Is somebody working on making Imperative::Backward use this API? this
would be a big change which I'm not aware of. And using an IR is of a
much bigger scope than the change I'm proposing here for example.

I think I'm having difficulty understanding what are the arguments
here. I'm saying I need to change one piece of my car and what you are
selling me is a new vehicle here?  Or your suggestion that we use
Relay for the graph passes in MXNet?

I would like to see C++ code examples, Python examples are not
sufficient when we talk about the core MXNet.

Pedro.






On Tue, May 14, 2019 at 5:39 PM Tianqi Chen <tqchen@cs.washington.edu> wrote:
>
> Thanks for the proposal. Let me share some of my thoughts:
>
> Specific comments on the proposal
> -----------------------------------------------
> The heavy use of generic in the Graph type was a huge departure from
> type-erased data structure which was presented in the previous design.
> While we understand the advantage of typed language(more compile-time
> checking) and type-erased types(more dynamism) the heavy use of
> the template will actually make the project solely C++ focused, making it
> hard to expose intermediate(templatized) data structure to
> other languages like python/scala/clojure.
>
> While I fully understand some of the lessons taught in programming
> C++(reduce shared_ptr, more typing etc.)
> We need to think about the context of MXNet project and **the need to
> support multi-language as a first-class**.
> Some of the type-erased types are design trade-offs made to support these
> features, and we need to think more
> carefully instead of just applying "rules for C++" which may bring problems.
>
> Future of NNVM
> ----------------------
> Given that this thread touched upon what we should do for better
> computational graph handling. I would recommend also to take a look at
> NNVMv2 -- relay.
>
> Relay addresses many of the wish-lists in the proposal already, such as
> operator fusion, high order gradient, offload to hardware, isolated
> compilation, deployment on edge and accelerators etc.
> Relay also address problems not yet being mentioned in the proposal,
> including control flow and dynamic runtime, automatic layout optimization
> etc.
>
> Tianqi
>
> On Tue, May 14, 2019 at 5:06 PM Sheng Zha <zhasheng@apache.org> wrote:
>
> > Hi Pedro,
> >
> > Thanks for taking the inititaive. Skimming through the design doc, I
> > didn't see comparison with existing solutions such as relay in tvm, which
> > is already a dependency of mxnet already. Could you elaborate on comparison
> > with existing solutions in the design doc too?
> >
> > -sz
> >
> > On 2019/05/14 23:49:30, Pedro Larroy <pedro.larroy.lists@gmail.com>
> > wrote:
> > > Hi dev@
> > >
> > > As a result of my deep dives on the graph machinery I have created a
> > > new proposal to improve the operator graph in MXNet.
> > >
> > > This would mean superseding the use of NNVM Graph in MXNet and having
> > > a new implementation that we can use to simplify a lot of code and do
> > > powerful graph manipulation and passes such as operator fusion and
> > > other optimizations.
> > >
> > > As it would be a change with big impact and ramifications, your
> > > thoughts and feedback on the document would be highly appreciated so
> > > we can take potential future interesting use cases:
> > >
> > >
> > https://cwiki.apache.org/confluence/display/MXNET/MXVM%3A+Operator+graph+2.0
> > >
> > > Pedro.
> > >
> >

Mime
View raw message