mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Can Balioglu <cbalio...@fastmail.com>
Subject Re: Following semantic versioning
Date Sat, 10 Mar 2018 03:29:50 GMT
We found a rather "hacky" way to dynamically inject our operators and data iterators during
runtime. I can share the details if you are interested. We also plan to send out a design
proposal for a proper extensibility framework similar to TensorFlow's plugin API.

-Can

On Wed, Mar 7, 2018, at 00:28, Eric Xie wrote:
> One potential problem is with libstdc++. Specifically with vectors. Our 
> operator interface uses vectors, which breaks when core lib and 
> extension are compiled with different compiler version
> 
> On 2018/03/06 22:45:16, Chris Olivier <cjolivier01@gmail.com> wrote: 
> > The static init of your module should register your operator just as it
> > does for the operators in mxnet (NNVM_REGISTER_OP).  While I haven't done
> > it personally, I see no reason why it wouldn't work like any other operator
> > at that point.
> > 
> > On Tue, Mar 6, 2018 at 1:28 PM, Barber, Christopher <
> > Christopher.Barber@analog.com> wrote:
> > 
> > > There are two separate issues: how to compile against the MXNet source and
> > > how to dynamically load external dynamic libraries. While it would be nice
> > > to have all necessary headers in the same place, it doesn't really matter
> > > that much if people building extensions have to have access to the entire
> > > MXNet source tree. The real issue is whether you support the ability to
> > > dynamically load such extensions.
> > >
> > > - Christopher
> > >
> > > On 3/6/18, 4:12 PM, "Chris Olivier" <cjolivier01@gmail.com> wrote:
> > >
> > >     This was discussed in the past and so far, it seems possible for Unix
> > >     builds, although it’s going to be messy since the compile would
> > > generally
> > >     need access to all a large portion of the headers in the source tree,
> > > since
> > >     the “things needed to create your own op†aren’t necessarily
all
> > > contained
> > >     in the include/mxnet directory.
> > >
> > >     On Tue, Mar 6, 2018 at 11:40 AM kellen sunderland <
> > >     kellen.sunderland@gmail.com> wrote:
> > >
> > >     > Could we actually just define a mechanism so the libs could register
> > > their
> > >     > ops at runtime?  No linking required?
> > >     >
> > >     > On Tue, Mar 6, 2018, 8:36 PM Pedro Larroy <
> > > pedro.larroy.lists@gmail.com>
> > >     > wrote:
> > >     >
> > >     > > This is a good point. What additional blockers would there be
for
> > > linking
> > >     > > against a user provided library with custom operators?
> > >     > >
> > >     > >
> > >     > >
> > >     > > On Tue, Mar 6, 2018 at 5:16 PM, Barber, Christopher <
> > >     > > Christopher.Barber@analog.com> wrote:
> > >     > >
> > >     > > > To avoid this kind of problem, you really need to support
> > > features that
> > >     > > > allow MXNet to be extended without having to resort to forking.
> > > There
> > >     > is
> > >     > > > currently no way to add C++ custom operators without forking,
> > > and no
> > >     > way
> > >     > > to
> > >     > > > share such operators across projects. This creates a perverse
> > > incentive
> > >     > > to
> > >     > > > try to get changes that may not belong into the main product.
> > >     > > >
> > >     > > > On 3/5/18, 6:26 PM, "Marco de Abreu" <
> > > marco.g.abreu@googlemail.com>
> > >     > > > wrote:
> > >     > > >
> > >     > > >     Hello,
> > >     > > >
> > >     > > >     we recently had a few cases in which it has been attempted
> > > to add
> > >     > new
> > >     > > >     functionality to old branches because a customer of
somebodys
> > >     > > $DAY_JOB
> > >     > > >     requested it and was unable to switch to the latest
release
> > > or that
> > >     > > > certain
> > >     > > >     feature did not make it into the release. This lead
to quite
> > > a lot
> > >     > of
> > >     > > >     discussions and there was no clear standing within the
> > > community.
> > >     > > >
> > >     > > >     Just to cite semantic versioning:
> > >     > > >
> > >     > > >        1. MAJOR version when you make incompatible API changes,
> > >     > > >        2. MINOR version when you add functionality in a
> > >     > > > backwards-compatible
> > >     > > >        manner, and
> > >     > > >        3. PATCH version when you make backwards-compatible
bug
> > > fixes.
> > >     > > >
> > >     > > >
> > >     > > >     We as a community agreed on following this system and
I
> > > think we
> > >     > > should
> > >     > > >     either stick to it or drop it entirely - exceptions
to
> > > SemVer are
> > >     > > > usually
> > >     > > >     discouraged. While I see that adding functionality might
be
> > > a minor
> > >     > > > thing,
> > >     > > >     I don't think that we should educate our users into
> > > expecting us to
> > >     > > >     backport new features. The development happening on
the
> > > Apache
> > >     > MXNet
> > >     > > >     repository should not be influenced by something like
this;
> > >     > > especially
> > >     > > >     considering that whoever supports that customer on their
> > > $DAY_JOB
> > >     > can
> > >     > > >     assist them at creating a fork and cherrypicking that
> > > feature. But
> > >     > I
> > >     > > > don't
> > >     > > >     see much reason why we're running against best pracitices.
> > > One
> > >     > > > important
> > >     > > >     thing to note here is that we're not maintaining CI
on old
> > > branches
> > >     > > and
> > >     > > >     neither are we making patch releases - so what do these
> > > users do?
> > >     > > > Compile
> > >     > > >     off a stale development branch with unvalidated changes?
> > >     > > >
> > >     > > >     In my opinion this whole topic is just causing trouble
and
> > >     > > > fragementation
> > >     > > >     on our end. If a features doesn't make it into the release
> > > (for
> > >     > > > whatever
> > >     > > >     reason), so be it. But I think we should discuss this
topic
> > > and
> > >     > > > emphasize a
> > >     > > >     no-exceptions-rule to SemVer.
> > >     > > >
> > >     > > >     Best regards,
> > >     > > >     Marco
> > >     > > >
> > >     > > >
> > >     > > >
> > >     > >
> > >     >
> > >
> > >
> > >
> > 

Mime
View raw message