apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Weise <...@apache.org>
Subject Re: Java packages: legacy -> org.apache.apex
Date Mon, 21 Aug 2017 15:26:31 GMT
+ 1 for release-3 branch instead of release-3.8

I would suggest we vote on two proposals (4.x as well as the option to
change names and start at 1.0). Any other suggestions? I'm planning to
start a VOTE thread by EOD.

Thanks,
Thomas


On Sat, Aug 19, 2017 at 1:56 PM, Pramod Immaneni <pramod@datatorrent.com>
wrote:

> I think if there is a will to allow older apps to continue to benefit from
> improvements to operators, without changes, it can be done along with the
> proposed move, especially since we are not taking about all operators.
>
> However, since that is not the case, I am fine with starting a 4.0 release
> line as long as we don't seal the future of 3.x now. What I would prefer is
> to create a release-3 branch and not a release-3.8 branch now and let the
> future determine if 3.8 will be the last 3.x minor release.
>
> Thanks
>
> On Fri, Aug 18, 2017 at 5:04 PM Thomas Weise <thw@apache.org> wrote:
>
> > The proposed change does NOT require changes to existing apps. It follows
> > what you see for example in the JDK where older classes are retired
> without
> > making breaking major release changes.
> >
> > Even though not required technically, it looks like the preference is to
> > make this change with a 4.0 release. I will therefore propose to change
> the
> > master branch to 4.0-SNAPSHOT and make this and other changes deemed
> > worthwhile.
> >
> > To me it is more important that to newcomers the codebase does not look
> > like an outdated legacy situation and therefore if I have to make a
> choice
> > than pretty much any version number will do for that. So far I have seen
> > very little interest and initiative in addressing maintenance work like
> the
> > packages, checkstyle, CI stability, documentation etc. So it would be
> great
> > if people interested would step up and contribute.
> >
> > Here is the next proposal how to proceed:
> >
> >    - create release-3.8 branch for last 3.8 release right away
> >    - update master to 4.0-SNAPSHOT as part of the open PR #664
> >    - identify further cleanup work for master
> >    - release 4.0
> >
> > Thanks,
> > Thomas
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Thu, Aug 17, 2017 at 4:48 PM, Amol Kekre <amol@datatorrent.com>
> wrote:
> >
> > > This following pull request should be taken up in 4.0.0. See my
> comments
> > in
> > > https://github.com/apache/apex-malhar/pull/664
> > > <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%
> > > 2Fapache%2Fapex-malhar%2Fpull%2F664&sa=D&ust=1503020269444000&usg=
> > > AFQjCNHDPZIg2e6I33Jb1XjB5Ir1FkdURQ>
> > > https://github.com/apache/apex-malhar/pull/662
> > > <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%
> > > 2Fapache%2Fapex-malhar%2Fpull%2F662&sa=D&ust=1503020269444000&usg=
> > > AFQjCNF7Xde7X55M3qmc8z-D5y3ZwNg7Fg>
> > >
> > > This merge should not be done without a consensus. This will require
> code
> > > changes to existing apps.
> > >
> > > Thks,
> > > Amol
> > >
> > >
> > >
> > > E:amol@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre*
> > >
> > > www.datatorrent.com
> > >
> > >
> > > On Mon, Aug 14, 2017 at 7:48 PM, Thomas Weise <thw@apache.org> wrote:
> > >
> > > > Hi,
> > > >
> > > > I opened following PRs for the package change:
> > > >
> > > > https://github.com/apache/apex-malhar/pull/662
> > > >
> > > > Moves all classes with history retained (hence 2 commits). Also
> > contains
> > > > checkstyle and other mechanical changes.
> > > >
> > > > https://github.com/apache/apex-malhar/pull/664
> > > >
> > > > Adds backward compatibility jar.
> > > >
> > > > Once above PRs are merged the new artifact can be deployed and
> > introduced
> > > > as dependency in malhar-library.
> > > >
> > > > Please review.
> > > >
> > > > Thanks,
> > > > Thomas
> > > >
> > > >
> > > >
> > > > On Sun, Jul 16, 2017 at 7:04 AM, Thomas Weise <thw@apache.org>
> wrote:
> > > >
> > > > > My original list of work items contained the b/w compatibility
> > aspect,
> > > I
> > > > > don't think there should be any confusion of whether it will be
> > covered
> > > > > here or not.
> > > > >
> > > > > The proposed shading will provide the old classes and they will be
> > > frozen
> > > > > as of release 3.7. That's the same as making a copy of the code and
> > > never
> > > > > again making changes to the original classes. This cannot be
> > > accomplished
> > > > > by using the older 3.7 release in your project because you cannot
> > use 2
> > > > > different versions of Malhar in parallel unless you apply shading.
> > > > >
> > > > > The shaded artifact will only expose the com.datatorrent classes,
> and
> > > > they
> > > > > will be self-contained as the rest of the classes that they may
> > depend
> > > on
> > > > > are shaded. The shaded artifact does not evolve, there are not more
> > > > changes
> > > > > to com.datatorrent classes after they are relocated in master.
> > > > >
> > > > > Thanks,
> > > > > Thomas
> > > > >
> > > > >
> > > > > On Sun, Jul 16, 2017 at 2:00 AM, Pramod Immaneni <
> > > pramod@datatorrent.com
> > > > >
> > > > > wrote:
> > > > >
> > > > >> I don't think we can limit the topic strictly to relocation
> without
> > > > having
> > > > >> a good b/w compatibility story or at least one that goes far
> enough.
> > > > >>
> > > > >> The shading idea sounds interesting. Why not let the shaded
> version
> > > move
> > > > >> forward with each release till we hit a major release. If it is
> > going
> > > to
> > > > >> remain pegged at 3.7.0, why shade in the first place as the
> regular
> > > > 3.7.0
> > > > >> release would do the same job and it would be same as the loss of
> > b/w
> > > > >> compatibility with newer releases.
> > > > >>
> > > > >> Thanks
> > > > >>
> > > > >> On Sat, Jul 15, 2017 at 7:57 AM, Thomas Weise <thw@apache.org>
> > wrote:
> > > > >>
> > > > >> > Discussing what in the future might become stable needs to be a
> > > > separate
> > > > >> > thread, it will be a much bigger discussion.
> > > > >> >
> > > > >> > The topic here is to relocate the packages. With a few
> exceptions
> > > > >> > relocation won't affect the semantic versioning. Semantic
> > versioning
> > > > is
> > > > >> > essentially not effective for Malhar because almost everything
> is
> > > > >> @Evolving
> > > > >> > (and there are reasons for that.. -> separate topic)
> > > > >> >
> > > > >> > I don't really like the idea of creating bw compatibility stubs
> > for
> > > > the
> > > > >> > follow up PR. It creates even more clutter in the source tree
> than
> > > > there
> > > > >> > already is and so here is an alternative suggestion:
> > > > >> >
> > > > >> > https://github.com/tweise/apex-malhar/blob/malhar37-
> > > > >> > compat/shaded-malhar37/pom.xml
> > > > >> >
> > > > >> > Create a shaded artifact that provides the old com.datatorrent.*
> > > > >> classes as
> > > > >> > of release 3.7. Users can include that artifact if they don't
> want
> > > to
> > > > >> > change import statements. At the same time they have an
> incentive
> > to
> > > > >> switch
> > > > >> > to the relocated classes to take advantage of bug fixes and new
> > > > >> > functionality.
> > > > >> >
> > > > >> > I will work on the first PR that does the relocate. In the
> > meantime,
> > > > we
> > > > >> can
> > > > >> > finalize what backward compatibility support we want to provide
> > and
> > > > how.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Thomas
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Fri, Jul 14, 2017 at 11:33 AM, Pramod Immaneni <
> > > > >> pramod@datatorrent.com>
> > > > >> > wrote:
> > > > >> >
> > > > >> > > How about coming up with a list of @Evolving operators that we
> > > would
> > > > >> like
> > > > >> > > to see in the eventual stable list and move those along with
> the
> > > not
> > > > >> > > @Evolving ones in org.apache.apex with b/w stubs and leave the
> > > rest
> > > > as
> > > > >> > they
> > > > >> > > are. Then have a follow up JIRA for the rest to be moved over
> to
> > > > >> contrib
> > > > >> > > and be deprecated.
> > > > >> > >
> > > > >> > > Thanks
> > > > >> > >
> > > > >> > > On Fri, Jul 14, 2017 at 10:37 AM, Thomas Weise <
> > > > >> thomas.weise@gmail.com>
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > We need to keep the discussion here on topic, if other
> things
> > > are
> > > > >> piled
> > > > >> > > on
> > > > >> > > > top then nothing gets done.
> > > > >> > > >
> > > > >> > > > Most operators today are not stable, they are @Evolving. So
> > > > perhaps
> > > > >> it
> > > > >> > is
> > > > >> > > > necessary to have a separate discussion about that, outside
> of
> > > > this
> > > > >> > > thread.
> > > > >> > > > My guess is that there are only a few operators that we
> could
> > > > >> declare
> > > > >> > > > stable. Specifically, under contrib the closest one might
> have
> > > > been
> > > > >> > > Kafka,
> > > > >> > > > but that is already superseded by the newer versions.
> > > > >> > > >
> > > > >> > > > Thomas
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > On Fri, Jul 14, 2017 at 10:21 AM, Pramod Immaneni <
> > > > >> > > pramod@datatorrent.com>
> > > > >> > > > wrote:
> > > > >> > > >
> > > > >> > > > > We had a discussion a while back, agreed to relegate
> > > non-stable
> > > > >> and
> > > > >> > > > > experimental operators to contrib and also added this to
> > > > >> contribution
> > > > >> > > > > guidelines. We aexecuted on this and cleaned up the repo
> by
> > > > moving
> > > > >> > > > > operators deemed non-suitable for production use at that
> > time
> > > to
> > > > >> > > contrib.
> > > > >> > > > > So I wouldn't say the operators that are at the top level
> > > today
> > > > or
> > > > >> > the
> > > > >> > > > ones
> > > > >> > > > > in library are 0.x.x quality. Granted, we may need to do
> one
> > > > more
> > > > >> > pass
> > > > >> > > as
> > > > >> > > > > some of the operators may no longer be considered the
> right
> > > > >> > > > implementations
> > > > >> > > > > with the advent of the windowed operator.
> > > > >> > > > >
> > > > >> > > > > Since contrib used to be the place that housed operators
> > that
> > > > >> > required
> > > > >> > > > > third party libraries, there are some production quality
> > > > >> operators in
> > > > >> > > > there
> > > > >> > > > > that need to be pulled out into top level modules.
> > > > >> > > > >
> > > > >> > > > > I think we are in agreement that for operators that we
> > > consider
> > > > >> > stable,
> > > > >> > > > we
> > > > >> > > > > should provide a b/w stub. I would add that we strongly
> > > consider
> > > > >> > > creating
> > > > >> > > > > the org.apache.apex counterparts of any stable operators
> > that
> > > > are
> > > > >> in
> > > > >> > > > > contrib out in top level modules and have the
> > com.datatorrent
> > > > >> stubs
> > > > >> > in
> > > > >> > > > > contrib.
> > > > >> > > > >
> > > > >> > > > > For the operators not considered stable, I would prefer we
> > > > either
> > > > >> > leave
> > > > >> > > > the
> > > > >> > > > > current package path as is or if we change the package
> path
> > > then
> > > > >> > create
> > > > >> > > > the
> > > > >> > > > > b/w stub as I am not sure how widely they are in use (so,
> in
> > > > >> essence,
> > > > >> > > > > preserve semantic versioning). It would be good if there
> is
> > a
> > > > >> > followup
> > > > >> > > > JIRA
> > > > >> > > > > that takes a look at what other operators can be moved to
> > > > contrib
> > > > >> > with
> > > > >> > > > the
> > > > >> > > > > advent of the newer frameworks and understanding.
> > > > >> > > > >
> > > > >> > > > > Thanks
> > > > >> > > > >
> > > > >> > > > > On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise <
> > thw@apache.org
> > > >
> > > > >> > wrote:
> > > > >> > > > >
> > > > >> > > > > > Most of the operators evolve, as is quite clearly
> > indicated
> > > by
> > > > >> > > > @Evolving
> > > > >> > > > > > annotations. So while perhaps 0.x.x would be a more
> > > > appropriate
> > > > >> > > version
> > > > >> > > > > > number, I don't think you can change that.
> > > > >> > > > > >
> > > > >> > > > > > Thomas
> > > > >> > > > > >
> > > > >> > > > > > On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov <
> > > > >> > v.rozov@datatorrent.com
> > > > >> > > >
> > > > >> > > > > > wrote:
> > > > >> > > > > >
> > > > >> > > > > > > If entire library is not stable, its version should be
> > > 0.x.x
> > > > >> and
> > > > >> > > > there
> > > > >> > > > > > > should not be any semantic versioning enabled or
> > implied.
> > > It
> > > > >> > > evolves.
> > > > >> > > > > If
> > > > >> > > > > > > some operators are stable as 3.8.x version suggests,
> the
> > > > >> library
> > > > >> > > > should
> > > > >> > > > > > > follow semantic versioning and it is not OK to make a
> > > major
> > > > >> > change
> > > > >> > > > that
> > > > >> > > > > > > breaks semantic versioning across the entire library.
> It
> > > is
> > > > >> not a
> > > > >> > > > > finding
> > > > >> > > > > > > an excuse not to make a change. For me semantic
> > versioning
> > > > and
> > > > >> > > > backward
> > > > >> > > > > > > compatibility is more important than a proper package
> > > name.
> > > > >> > > > > > >
> > > > >> > > > > > > Thank you,
> > > > >> > > > > > >
> > > > >> > > > > > > Vlad
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > > On 7/14/17 09:11, Thomas Weise wrote:
> > > > >> > > > > > >
> > > > >> > > > > > >> Semantic versioning makes sense when you have a
> stable
> > > > >> baseline.
> > > > >> > > As
> > > > >> > > > > long
> > > > >> > > > > > >> as
> > > > >> > > > > > >> frequent fixes need to be made to address basic
> issues,
> > > it
> > > > >> makes
> > > > >> > > no
> > > > >> > > > > > sense
> > > > >> > > > > > >> to declare operators stable, which is why they are
> > marked
> > > > >> > > evolving.
> > > > >> > > > > > >>
> > > > >> > > > > > >> Instead of finding reasons to avoid changes that
> should
> > > > have
> > > > >> > been
> > > > >> > > > > made a
> > > > >> > > > > > >> long time ago, why not discuss what a "stable
> operator"
> > > is
> > > > >> and
> > > > >> > > which
> > > > >> > > > > > ones
> > > > >> > > > > > >> deserve to be in that category. Those are the ones
> that
> > > > will
> > > > >> get
> > > > >> > > the
> > > > >> > > > > > >> backward compatibility stub.
> > > > >> > > > > > >>
> > > > >> > > > > > >> Thanks,
> > > > >> > > > > > >> Thomas
> > > > >> > > > > > >>
> > > > >> > > > > > >>
> > > > >> > > > > > >> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <
> > > > >> > > > v.rozov@datatorrent.com>
> > > > >> > > > > > >> wrote:
> > > > >> > > > > > >>
> > > > >> > > > > > >> My preference is to agree that the next Malhar
> release
> > is
> > > > >> 4.0.0
> > > > >> > > and
> > > > >> > > > > make
> > > > >> > > > > > >>> the proposed changes when 3.8.0 goes out. Otherwise
> > why
> > > to
> > > > >> keep
> > > > >> > > > > > semantic
> > > > >> > > > > > >>> versioning checks on Malhar in case there is no
> > version
> > > > >> > > > > compatibility.
> > > > >> > > > > > >>>
> > > > >> > > > > > >>> Thank you,
> > > > >> > > > > > >>>
> > > > >> > > > > > >>> Vlad
> > > > >> > > > > > >>>
> > > > >> > > > > > >>>
> > > > >> > > > > > >>> On 7/14/17 08:11, Thomas Weise wrote:
> > > > >> > > > > > >>>
> > > > >> > > > > > >>> next release is 3.8.0
> > > > >> > > > > > >>>>
> > > > >> > > > > > >>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <
> > > > >> > > > > v.rozov@datatorrent.com>
> > > > >> > > > > > >>>> wrote:
> > > > >> > > > > > >>>>
> > > > >> > > > > > >>>> next release - 3.9.0 or 4.0.0?
> > > > >> > > > > > >>>>
> > > > >> > > > > > >>>>> Thank you,
> > > > >> > > > > > >>>>>
> > > > >> > > > > > >>>>> Vlad
> > > > >> > > > > > >>>>>
> > > > >> > > > > > >>>>> On 7/13/17 22:25, Thomas Weise wrote:
> > > > >> > > > > > >>>>>
> > > > >> > > > > > >>>>> It is time to resurrect this thread and get going
> > with
> > > > the
> > > > >> > > work.
> > > > >> > > > > > >>>>>
> > > > >> > > > > > >>>>>> For the next release, I will sign up to do the
> > > package
> > > > >> move
> > > > >> > in
> > > > >> > > > > > Malhar:
> > > > >> > > > > > >>>>>>
> > > > >> > > > > > >>>>>> https://issues.apache.org/
> > > jira/browse/APEXMALHAR-2517
> > > > >> > > > > > >>>>>>
> > > > >> > > > > > >>>>>> In general this will be straightforward; most
> > classes
> > > > in
> > > > >> > > Malhar
> > > > >> > > > > are
> > > > >> > > > > > >>>>>> marked
> > > > >> > > > > > >>>>>> evolving and it is trivial for users to change
> > import
> > > > >> > > > statements.
> > > > >> > > > > > >>>>>> However,
> > > > >> > > > > > >>>>>> I would suggest to discuss if there are selected
> > > > >> operators
> > > > >> > > that
> > > > >> > > > > are
> > > > >> > > > > > >>>>>> worth
> > > > >> > > > > > >>>>>> keeping a backward compatibility stub in the old
> > > > >> location.
> > > > >> > > > > > >>>>>>
> > > > >> > > > > > >>>>>> Here is my plan:
> > > > >> > > > > > >>>>>>
> > > > >> > > > > > >>>>>> 1. Relocate all classes in *lib* and *contrib*
> > within
> > > > one
> > > > >> > PR -
> > > > >> > > > > this
> > > > >> > > > > > is
> > > > >> > > > > > >>>>>> all
> > > > >> > > > > > >>>>>>
> > > > >> > > > > > >>>>>> IDE automated work
> > > > >> > > > > > >>>>>> 2. Add backward compatibility classes, if, any in
> > > > >> separate
> > > > >> > PR
> > > > >> > > > > > >>>>>> 3. Create PR for Megh library to reflect moved
> > > classes
> > > > >> > > > > > >>>>>>
> > > > >> > > > > > >>>>>> Thanks,
> > > > >> > > > > > >>>>>> Thomas
> > > > >> > > > > > >>>>>>
> > > > >> > > > > > >>>>>>
> > > > >> > > > > > >>>>>>
> > > > >> > > > > > >>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <
> > > > >> > > > > > >>>>>> pramod@datatorrent.com
> > > > >> > > > > > >>>>>> wrote:
> > > > >> > > > > > >>>>>>
> > > > >> > > > > > >>>>>> Inline
> > > > >> > > > > > >>>>>>
> > > > >> > > > > > >>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <
> > > > >> > > thw@apache.org>
> > > > >> > > > > > >>>>>>> wrote:
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>> -->
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod
> Immaneni <
> > > > >> > > > > > >>>>>>>> pramod@datatorrent.com
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> wrote:
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> For malhar, for existing operators, I prefer we
> > do
> > > > >> this as
> > > > >> > > > part
> > > > >> > > > > of
> > > > >> > > > > > >>>>>>>> the
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> planned refactoring for breaking the monolith
> > > modules
> > > > >> into
> > > > >> > > > baby
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>> packages
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>> and would also prefer deprecating the existing
> > > > >> operators
> > > > >> > in
> > > > >> > > > > place.
> > > > >> > > > > > >>>>>>>> Refactor into smaller modules was discussed for
> > > > >> > > malhar-contrib
> > > > >> > > > > and
> > > > >> > > > > > >>>>>>>> given
> > > > >> > > > > > >>>>>>>> the overall state of that module I think it is
> OK
> > > to
> > > > >> defer
> > > > >> > > > > package
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> renaming
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>> there. I do however prefer to see the package
> > rename
> > > > >> > > addressed
> > > > >> > > > > for
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>>> other
> > > > >> > > > > > >>>>>>>> modules, especially for the main library
> module.
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> Should we consider breaking the library into
> > > smaller
> > > > >> > modules
> > > > >> > > > as
> > > > >> > > > > > >>>>>>>> well,
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> the
> > > > >> > > > > > >>>>>>> file/block operators for example probably can be
> > in
> > > > >> their
> > > > >> > own
> > > > >> > > > > > module
> > > > >> > > > > > >>>>>>> from
> > > > >> > > > > > >>>>>>> just an organizational perspective.
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>> This
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>>> will help us achieve two things. First, the
> user
> > > will
> > > > >> see
> > > > >> > > all
> > > > >> > > > > the
> > > > >> > > > > > >>>>>>>>> new
> > > > >> > > > > > >>>>>>>>> changes at once as opposed to dealing with it
> > > twice
> > > > >> (with
> > > > >> > > > > package
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>> rename
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>> and dependency changes) and second it will
> allow
> > > for
> > > > a
> > > > >> > > > smoother
> > > > >> > > > > > >>>>>>>> transition
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> as the existing code will still work in a
> > > deprecated
> > > > >> > state.
> > > > >> > > It
> > > > >> > > > > > will
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>>> also
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>> give a more consistent structure to malhar. For
> > new
> > > > >> > > operators,
> > > > >> > > > > we
> > > > >> > > > > > >>>>>>>> can
> > > > >> > > > > > >>>>>>>> go
> > > > >> > > > > > >>>>>>>> with the new package path but we need to ensure
> > > they
> > > > >> will
> > > > >> > > get
> > > > >> > > > > > moved
> > > > >> > > > > > >>>>>>>> into
> > > > >> > > > > > >>>>>>>> the baby packages as well.
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> I think existing operators should be renamed so
> > > that
> > > > >> git
> > > > >> > > > history
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>> remains. A
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>> possible solution for backward compatibility
> could
> > > be
> > > > to
> > > > >> > > > > > subsequently
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>>> add
> > > > >> > > > > > >>>>>>>> empty subclasses in the previous location (for
> > > > existing
> > > > >> > > > concrete
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> operators
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>> that we know are actually in use) to simplify
> > > > migration
> > > > >> for
> > > > >> > > > > users.
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> Yes we can do that.
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> For demos, we can modify the paths as the apps
> > are
> > > > >> > typically
> > > > >> > > > > used
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>> wholesale
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> and the interface is typically manual
> > interaction.
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>>> For core, if we are adding new api subsystems,
> > > like
> > > > >> the
> > > > >> > > > > launcher
> > > > >> > > > > > >>>>>>>>> api
> > > > >> > > > > > >>>>>>>>> we
> > > > >> > > > > > >>>>>>>>> added recently for example, we can go with new
> > > > package
> > > > >> > path
> > > > >> > > > but
> > > > >> > > > > > if
> > > > >> > > > > > >>>>>>>>> we
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>> are
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>> making incremental additions to existing
> > > > >> functionality, I
> > > > >> > > feel
> > > > >> > > > > it
> > > > >> > > > > > is
> > > > >> > > > > > >>>>>>>> better
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> to keep it in the same package. I also prefer
> we
> > > keep
> > > > >> the
> > > > >> > > > > package
> > > > >> > > > > > of
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>>> the
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>> implementation classes consistent with api, for
> > > > >> > > > > understandability
> > > > >> > > > > > >>>>>>>> and
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> readability of the code. So, for example, we
> > don't
> > > > >> change
> > > > >> > > > > package
> > > > >> > > > > > >>>>>>>>> path
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>> of
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>> LogicalPlan as it is an implementation of DAG.
> It
> > > is
> > > > >> > > > subjective,
> > > > >> > > > > > but
> > > > >> > > > > > >>>>>>>> it
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> will be good if we can also do the same with
> > > classes
> > > > >> > closely
> > > > >> > > > > > related
> > > > >> > > > > > >>>>>>>>> to
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>> the
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>> implementation classes as well. Maybe we can
> > moving
> > > > >> these
> > > > >> > > on a
> > > > >> > > > > > >>>>>>>> package
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>>> by
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>> package basis, like everything in
> > > > >> > > com.datatorrent.stram.engine
> > > > >> > > > > > could
> > > > >> > > > > > >>>>>>>> be
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> moved. For completely internal components like
> > > buffer
> > > > >> > > server,
> > > > >> > > > we
> > > > >> > > > > > can
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>> move
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>> them wholesale. We can consider moving all api
> > and
> > > > >> > classes,
> > > > >> > > > when
> > > > >> > > > > > we
> > > > >> > > > > > >>>>>>>> go
> > > > >> > > > > > >>>>>>>> to
> > > > >> > > > > > >>>>>>>> next major release but would like to see if we
> > can
> > > > >> find a
> > > > >> > > way
> > > > >> > > > to
> > > > >> > > > > > >>>>>>>> support
> > > > >> > > > > > >>>>>>>> existing api for one more major release in
> > > deprecated
> > > > >> > mode.
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> The point of the major release is to enable
> > > backward
> > > > >> > > > > incompatible
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>> changes
> > > > >> > > > > > >>>>>>>> and I don't think it is realistic to support
> the
> > > > >> existing
> > > > >> > > API
> > > > >> > > > > for
> > > > >> > > > > > >>>>>>>> another
> > > > >> > > > > > >>>>>>>> major release. IMO it is also not necessary as
> > most
> > > > >> > existing
> > > > >> > > > > > >>>>>>>> application
> > > > >> > > > > > >>>>>>>> code refers to operators, attributes and the
> > > > >> application
> > > > >> > > > > > interface.
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> Perhaps
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>> it is possible to keep those around as interface
> > > > >> extensions
> > > > >> > > to
> > > > >> > > > > help
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>>> migration. Custom operators may need to be
> > migrated
> > > > to
> > > > >> > > reflect
> > > > >> > > > > API
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> changes,
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>> and I would consider that a reasonable task for
> > > > operator
> > > > >> > > > > developers
> > > > >> > > > > > >>>>>>> as
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>>> part
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>> of a major upgrade.
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>>> It would be good if we can keep them as
> > deprecated
> > > > >> > interface
> > > > >> > > > > > >>>>>>>> extensions
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> for
> > > > >> > > > > > >>>>>>> one release to provide a smoother transition.
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>> API and implementation in engine are kept
> separate
> > > > >> > > > intentionally.
> > > > >> > > > > > >>>>>>> They
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>> reside in different packages today, so I don't
> > see a
> > > > >> > problem
> > > > >> > > > > > renaming
> > > > >> > > > > > >>>>>>>> com.datatorrent.stram.engine as you say, even
> > when
> > > > the
> > > > >> API
> > > > >> > > > > cannot
> > > > >> > > > > > be
> > > > >> > > > > > >>>>>>>> touched right away.
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> They are different packages but sharing a
> common
> > > > prefix
> > > > >> > with
> > > > >> > > > api
> > > > >> > > > > > >>>>>>>> will
> > > > >> > > > > > >>>>>>>> be
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> helpful to someone new to codebase in terms of
> > > > >> > readability.
> > > > >> > > > Not
> > > > >> > > > > a
> > > > >> > > > > > >>>>>>> big
> > > > >> > > > > > >>>>>>> deal
> > > > >> > > > > > >>>>>>> and can be changed.
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>> Thanks
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise <
> > > > >> > > thw@apache.org>
> > > > >> > > > > > >>>>>>>> wrote:
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>>> Hi,
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>> This topic has come up on several PRs and I
> > think
> > > it
> > > > >> > > > warrants a
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>>> broader
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>> discussion.
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> At the time of incubation, the decision was to
> > > defer
> > > > >> > change
> > > > >> > > of
> > > > >> > > > > > Java
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>>> packages from com.datatorrent to
> > org.apache.apex
> > > > till
> > > > >> > next
> > > > >> > > > > major
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>>> release
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>> to
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>> ensure backward compatibility for users.
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>>> Unfortunately that has lead to some
> confusion,
> > as
> > > > >> > > > contributors
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>>> continue
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>> to
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> add new code under legacy packages.
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>>> It is also a wider issue that examples for
> > using
> > > > Apex
> > > > >> > > > continue
> > > > >> > > > > > to
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>>> refer
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>> to
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> com.datatorrent packages, nearly one year after
> > > > >> > graduation.
> > > > >> > > > More
> > > > >> > > > > > and
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>>> more
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>> user code is being built on top of something
> > that
> > > > >> needs
> > > > >> > to
> > > > >> > > > > > change,
> > > > >> > > > > > >>>>>>>>> the
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>> can
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> is being kicked down the road and users will
> face
> > > > more
> > > > >> > > changes
> > > > >> > > > > > >>>>>>>>> later.
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>>> I would like to propose the following:
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>>> 1. All new code has to be submitted under
> > > > >> > org.apache.apex
> > > > >> > > > > > packages
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>>> 2. Not all code is under backward
> compatibility
> > > > >> > > restriction
> > > > >> > > > > and
> > > > >> > > > > > in
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>>> those
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>> cases we can rename the packages right away.
> > > > Examples:
> > > > >> > > buffer
> > > > >> > > > > > >>>>>>>>> server,
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>> engine, demos/examples, benchmarks
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>>> 3. Discuss when the core API and operators
> can
> > be
> > > > >> > changed.
> > > > >> > > > For
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>>> operators
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>> we
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>> have a bit more freedom to do changes before a
> > > major
> > > > >> > > release
> > > > >> > > > as
> > > > >> > > > > > >>>>>>>>> most
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>>> of
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>> them are marked @Evolving and users have the
> > > ability
> > > > >> to
> > > > >> > > > > continue
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> using
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>> prior version of Malhar with newer engine due
> to
> > > > >> engine
> > > > >> > > > > backward
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> compatibility guarantee.
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>>> Thanks,
> > > > >> > > > > > >>>>>>>>>> Thomas
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message