From dev-return-2403-archive-asf-public=cust-asf.ponee.io@mxnet.incubator.apache.org Sun Mar 11 17:19:27 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 4641118060F for ; Sun, 11 Mar 2018 17:19:26 +0100 (CET) Received: (qmail 52935 invoked by uid 500); 11 Mar 2018 16:19:25 -0000 Mailing-List: contact dev-help@mxnet.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@mxnet.incubator.apache.org Delivered-To: mailing list dev@mxnet.incubator.apache.org Received: (qmail 52923 invoked by uid 99); 11 Mar 2018 16:19:24 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 11 Mar 2018 16:19:24 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id C812EC205D for ; Sun, 11 Mar 2018 16:19:23 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.429 X-Spam-Level: ** X-Spam-Status: No, score=2.429 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_REPLY=1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id DmejcNwD8Y2a for ; Sun, 11 Mar 2018 16:19:20 +0000 (UTC) Received: from mail-it0-f54.google.com (mail-it0-f54.google.com [209.85.214.54]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 5ED0D5F5E0 for ; Sun, 11 Mar 2018 16:19:19 +0000 (UTC) Received: by mail-it0-f54.google.com with SMTP id k79-v6so8412835ita.2 for ; Sun, 11 Mar 2018 09:19:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=ymUcedVv31B0BXPeXbpkuUp0n+2ktbtXNLUzZJ2tlzA=; b=R+zlTbbJAmjE1U0q5jVcL+ecgV3rJ7YrArrJfaJml2s7OMCdXmiNv+DpFjnAp3xpjL 8yRMg/g3546ihlgTBsP4BkXGsCcBKWuGv+L6UG2jp7SEqGaa1CZtspLLltiK1vNy1S7G QCyqXD9Sg49PJPogdVPuLXeGOZhF3mEKJkolGVH0VqKhGDyySi+KbFRGpSv2EwV1aRje rgi5ZfC40C8TKKyYfPFolM3+QP0SqU5edHavXz2EZGkPU3O+4NyzEB+mtxBU8DngR/YI Q87TxbEXM6rokuzoUkM8CPlFYAwLD2mzWP901bEuA4rvO8Arw2wOAdn/TfXWwQw/X5oT 7otA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=ymUcedVv31B0BXPeXbpkuUp0n+2ktbtXNLUzZJ2tlzA=; b=i4DYPlN33SoEVXhpH5buxV0NCNyxPJ2i4HX/8JwBKeC+3Ik67N6086VN5vdxnsnzXN aMCvxhQxlhTR0RyQrUnr1wkWMPDTw008Wp0PXo4q1i6jliA6XtoWfu437TfXMwDfWH3h 6z4uH5/t0ZL6I8ZNF1ZWcBpp7M+2ELTFNCfIWpa07Yjnsqv15QZOYyUIO2gI6mmcb5IW k0oyxi+v0/uN5ss53nWEUovUsymy17EzKQfP8I8NERwGRjvj7htpRpvyzi0tq/hv4nSY MhGAuhrDrdpOqSlHtPDtHbyqe2cszbHVcaKq1agT/X3inDs26AMe2QVmu7VTOYkQ7DbP 1wfQ== X-Gm-Message-State: AElRT7E6HpBsoGoIQt3ViB44xoKRN+cvafZDWGRwD8gksfL0fI6FvffU wcYryKCikMv96HWv9zIwhwOaEOp0AqdHBgz0D2g= X-Google-Smtp-Source: AG47ELvQ7HM0vhNQ5YCYNYZ1AGoB79h2sADIXBb26Y+EzkQQDfGhSLcfCt64ZJQsiKdDPrSVJ1QsRzrC/w9iI1oUTuA= X-Received: by 10.36.58.206 with SMTP id m197mr5375474itm.24.1520785151970; Sun, 11 Mar 2018 09:19:11 -0700 (PDT) MIME-Version: 1.0 References: <5aa4efb0.4ae1500a.2126d.f2b7@mx.google.com> In-Reply-To: From: Chris Olivier Date: Sun, 11 Mar 2018 16:19:01 +0000 Message-ID: Subject: Re: Publishing Scala Package/namespace change To: dev@mxnet.incubator.apache.org Content-Type: multipart/alternative; boundary="001a11441060c3520e0567256285" --001a11441060c3520e0567256285 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Ok, so why don=E2=80=99t we have two votes? 1) change namespace is a separate vote since it=E2=80=99s a code change and= has different voting rules (can be vetoed) 2) whether to disconnect non-C-API versioning from C-API versioning and have parallel versioning of all non-C APIs (process rule, so majority, I think is the rule, right?) -Chris On Sun, Mar 11, 2018 at 8:46 AM kellen sunderland < kellen.sunderland@gmail.com> wrote: > Sorry, the namespace should have been 'org.apache.mxnet' with the artifac= t > as 'mxnet-incubating'. > > On Sun, Mar 11, 2018 at 4:44 PM, kellen sunderland < > kellen.sunderland@gmail.com> wrote: > > > YiZhi, In general I agree that your points and examples are the ideal > > case, but in the MXNet situation there are some trade-offs we have to > > make. Let me try to specifically answer your points: > > > > "Do you mean we have different version for 'ml.dmlc' namespace and > > 'org.apache' namespace?" > > No I am not trying to saying that. I believe Marco, Naveen and I are al= l > > proposing we use a single org.apache.incubating.mxnet namespace moving > > forward, which would require a major version change to our product API > > under our current versioning scheme. Marco and I are proposing we appl= y > > this MV change _only_ to the scala package's API. > > > > "How to tell which Scala API version works with which MXNet core versio= n? > > By document?" > > Yes users will be able to tell via the website, release docs, maven > > package information, pom file, etc. > > > > "How many users will read the whole document and carefully pair the > > version id before they run into a strange error and give up?" > > They won't get a strange error, assuming we're talking about Scala user= s > > who are upgrading from a package with the same namespace they will rely > on > > the package manager to give them an update which should be painless. > > > > Secondly software developers understand that packages, not products, ha= ve > > versions. They know that these versions are used to communicate when > APIs > > are broken. There's examples of Apache packages doing this for package= s > > that include multiple interfaces, for example first-party modules > packaged > > with the HTTP server, or log4j's language bindings (arguably quite > similar > > to what Naveen is doing). > > > > While we can debate the right way to version packages, I think there's = a > > clear community decision here to get Naveen unblocked: > > > > (1) We continue semantically versioning across all APIs, meaning that > this > > change would get released with MXNet 2.*. > > (2) You version package interfaces semantically and have a compatible > > version mapping. > > (3) Status quo, we continue to release a Scala package as-is, breaking > > apache guidelines for artifact generation. > > (4) We rely on the namespace change itself to communicate a change in t= he > > interface. We don't consider this a major change. > > > > My (non-binding) preference would be for option 2. > > > > -Kellen > > > > On Sun, Mar 11, 2018 at 12:44 PM, Marco de Abreu < > > marco.g.abreu@googlemail.com> wrote: > > > >> Changing namespaces is one example of a required major version change, > but > >> there are more reasons like general refactoring or some deprecated API= s > >> just being hard to maintain. Things like these happen quite frequently > and > >> it's a problem every software project has to face and find a solution > for. > >> > >> Regarding ' How to tell which Scala API version works with which MXNet > >> core > >> version?': We could just bundle MXNet with the released API package as > we > >> do right now, but we would give each interface it's own version and > >> publish > >> them on their distribution platforms accordingly. Just an example: > >> >Scala-Package -> MXNet-Version > >> >> 1.0 -> 1.0 > >> >> 1.1 -> 1.1 > >> >> 2.0 -> 1.2 > >> >> 2.1 -> 1.3 > >> >> 3.0 -> 2.0 > >> > >> > R-Package -> MXNet-Version > >> >> 1.0 -> 1.0 > >> >> 2.0 -> 1.1 > >> >> 2.1 -> 1.2 > >> >> 2.2 -> 1.3 > >> >> 3.0 -> 2.0 > >> > >> This is always an N-to-1 mapping, while N being the versions of our AP= Is > >> and 1 the MXNet Core version. From MXNets versioning perspective, this > >> would then looking the following: > >> > MXNet-Version -> APIs > >> >> 1.0 -> Scala_1.0; R_1.0 > >> >> 1.1 -> Scala_1.1; R_2.0 > >> >> 1.2 -> Scala_2.0; R_2.1 > >> >> 1.3 -> Scala_2.1; R_2.2 > >> >> 2.0 -> Scala_3.0; R_3.0 > >> > >> This would give us the liberty to develop MXNet without restricting us > too > >> much - of course, major version increments will still have to be > >> considered > >> carefully. I don't think that this would harm transparency too much an= d > >> there's no need to write big documentation. > >> > >> -Marco > >> > >> > >> On Sun, Mar 11, 2018 at 12:16 PM, YiZhi Liu > wrote: > >> > >> > I have no idea how separating Scala API version can solve the > >> > 'compatibility' problem. Do you mean we have different version for > >> > 'ml.dmlc' namespace and 'org.apache' namespace? Do these two version= s > >> > have same behavior? How to tell which Scala API version works with > >> > which MXNet core version? By document? How many users will read the > >> > whole document and carefully pair the version id before they run int= o > >> > a strange error and give up? > >> > > >> > Moreover, changing namespace is an issue that is really rare and > >> > hardly happens. For other 'compatibility' problem, for example, the > >> > class/function definitions, should handle the compatibility itself. > >> > You'll never expect a project to have a different version for changi= ng > >> > 'calculate(int)' to 'calculate(float)', it should just add a new > >> > function 'calculate(float)'. > >> > > >> > Regarding 'In this case the Scala interface is clearly a separate > >> > entity from the C API.'. Everything can be seen as a separate entity= , > >> > the mxnet engine, the graph description, operators, python API, gluo= n > >> > API, etc. We should think carefully what we want to provide, and wha= t > >> > our users need. > >> > > >> > As an example, Apache Spark, still has SparkR (R API), PySpark (Pyth= on > >> > API), MLLib, GraphX ... as part of its release, and have the same > >> > version as Spark core as well as its Scala/Java API. > >> > > >> > 2018-03-10 23:58 GMT-08:00 kellen sunderland < > >> kellen.sunderland@gmail.com > >> > >: > >> > > +1 (non-binding) to what Marco is describing. +1 (non-binding) to > >> > getting the Scala bindings with the namespace change into Maven. > >> > > > >> > > The general best practice for SemVer, which is used by most projec= ts > >> > that employ SemVer, is to apply SemVer to the public APIs of package= s > >> that > >> > ship with your project. If you have several independent APIs this > could > >> > mean that they are versioned separately from each other, and from th= e > >> > overall project versioning mechanism. > >> > > > >> > > For example, the .NET Core library ships with a number of binaries= , > >> each > >> > with their own SemVerioned APIs. They also have a high-level, easy = to > >> > understand version for the package as a whole: > >> > https://docs.microsoft.com/en-us/dotnet/core/versions/. > >> > > > >> > > Nodesource has a good description of this: > >> http://nodesource.com/blog/ > >> > semver-a-primer/ > >> > > =E2=80=9CSemver is a scheme for interface versioning for the benef= it of > >> > interface consumers, thus if a tool has multiple interfaces, e.g. an > API > >> > and a CLI, these interfaces may evolve independent versioning.=E2=80= =9D > >> > > > >> > > SemVer at its core is a communication mechanism to inform develope= rs > >> of > >> > incompatibilities. In this case the Scala interface is clearly a > >> separate > >> > entity from the C API. I.e. changing the Scala namespace isn=E2=80= =99t going > to > >> > break C API users. It does not communicate anything useful to these > >> users > >> > if we up their major version in response to a Scala change, it simpl= y > >> > breaks compatibility. If we group all interfaces together, and > >> increment > >> > whenever any of them has a breaking change we=E2=80=99ll soon be at = MXNet > >> version > >> > 587. We=E2=80=99ll be forcing our users to check compatibility and = update > their > >> > dependency tracking constantly. The end result is that our users wi= ll > >> stop > >> > pulling in new versions of the library. > >> > > > >> > > What I would propose is that (1) we have a high-level SemVer syste= m > >> that > >> > tracks our C_API. This is the =E2=80=98MXNet=E2=80=99 version that = we generally refer > >> to > >> > and emphasize for our public releases. For each API we have an > >> independent > >> > versioning system that if we can, we fix to the MXNet version. When > it > >> > makes sense we version these APIs independently. So for example we > >> could > >> > have a MXNet 1.2 release that ships with a 2.0 Scala API / R API. > >> > > > >> > > In terms of Apache process I think shipping artifacts with a > >> non-Apache > >> > namespace is a bigger issue than whatever versioning conventions we > >> decide > >> > to use. > >> > > > >> > > -Kellen > >> > > > >> > > From: Carin Meier > >> > > Sent: Saturday, March 10, 2018 1:41 PM > >> > > To: dev@mxnet.incubator.apache.org > >> > > Cc: dev@mxnet.apache.org > >> > > Subject: Re: Publishing Scala Package/namespace change > >> > > > >> > > +1 as well. I'm actively developing a Clojure package for MXNet th= at > >> uses > >> > > the jars from the Scala package. > >> > > > >> > > - Carin > >> > > > >> > > On Fri, Mar 9, 2018 at 4:44 PM, YiZhi Liu > >> wrote: > >> > > > >> > >> +1 for changing the namespace asap. for the maven deploy, we can > have > >> > >> it build along with pip deployment. > >> > >> > >> > >> > >> > >> 2018-03-09 10:15 GMT-08:00 Naveen Swamy : > >> > >> > Hi Guys, > >> > >> > > >> > >> > I am working on MXNet Scala Inference APIs > >> > >> > along with > >> another > >> > >> > contributor Roshani. A while back I noticed that we haven't bee= n > >> > >> publishing > >> > >> > the scala package to Maven for a while now(last one being > v0.11.1a > >> > under > >> > >> > the dmlc namespace). > >> > >> > Currently users have to build the package manually and then use > it, > >> > this > >> > >> > hinders adoption and also is painful to build everything from > >> source. > >> > >> > > >> > >> > I also see that we haven't changed the namespace to org.apache > and > >> > >> instead > >> > >> > are still ml.dmlc namespace. > >> > >> > > >> > >> > I wanted to seek your opinion about changing the MXNet-Scala > >> package > >> > >> > namespace to org.apache for the Scala package and publish to > Maven > >> in > >> > the > >> > >> > upcoming release. I understand that this probably breaks the > Semver > >> > >> > semantics that is agreed upon, However I would like to point ou= t > >> that > >> > the > >> > >> > Scala package has never been published to maven as 1.0 under > >> > org.apache. > >> > >> > > >> > >> > Open to suggestions. > >> > >> > > >> > >> > Thanks, Naveen > >> > >> > >> > >> > >> > >> > >> > >> -- > >> > >> Yizhi Liu > >> > >> DMLC member > >> > >> Amazon Web Services > >> > >> Vancouver, Canada > >> > >> > >> > > > >> > > >> > > >> > > >> > -- > >> > Yizhi Liu > >> > DMLC member > >> > Amazon Web Services > >> > Vancouver, Canada > >> > > >> > > > > > --001a11441060c3520e0567256285--