apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pramod Immaneni <pra...@datatorrent.com>
Subject Re: Java packages: legacy -> org.apache.apex
Date Sun, 16 Jul 2017 09:00:16 GMT
I don't think we can limit the topic strictly to relocation without having
a good b/w compatibility story or at least one that goes far enough.

The shading idea sounds interesting. Why not let the shaded version move
forward with each release till we hit a major release. If it is going to
remain pegged at 3.7.0, why shade in the first place as the regular 3.7.0
release would do the same job and it would be same as the loss of b/w
compatibility with newer releases.

Thanks

On Sat, Jul 15, 2017 at 7:57 AM, Thomas Weise <thw@apache.org> wrote:

> Discussing what in the future might become stable needs to be a separate
> thread, it will be a much bigger discussion.
>
> The topic here is to relocate the packages. With a few exceptions
> relocation won't affect the semantic versioning. Semantic versioning is
> essentially not effective for Malhar because almost everything is @Evolving
> (and there are reasons for that.. -> separate topic)
>
> I don't really like the idea of creating bw compatibility stubs for the
> follow up PR. It creates even more clutter in the source tree than there
> already is and so here is an alternative suggestion:
>
> https://github.com/tweise/apex-malhar/blob/malhar37-
> compat/shaded-malhar37/pom.xml
>
> Create a shaded artifact that provides the old com.datatorrent.* classes as
> of release 3.7. Users can include that artifact if they don't want to
> change import statements. At the same time they have an incentive to switch
> to the relocated classes to take advantage of bug fixes and new
> functionality.
>
> I will work on the first PR that does the relocate. In the meantime, we can
> finalize what backward compatibility support we want to provide and how.
>
> Thanks,
> Thomas
>
>
>
>
> On Fri, Jul 14, 2017 at 11:33 AM, Pramod Immaneni <pramod@datatorrent.com>
> wrote:
>
> > How about coming up with a list of @Evolving operators that we would like
> > to see in the eventual stable list and move those along with the not
> > @Evolving ones in org.apache.apex with b/w stubs and leave the rest as
> they
> > are. Then have a follow up JIRA for the rest to be moved over to contrib
> > and be deprecated.
> >
> > Thanks
> >
> > On Fri, Jul 14, 2017 at 10:37 AM, Thomas Weise <thomas.weise@gmail.com>
> > wrote:
> >
> > > We need to keep the discussion here on topic, if other things are piled
> > on
> > > top then nothing gets done.
> > >
> > > Most operators today are not stable, they are @Evolving. So perhaps it
> is
> > > necessary to have a separate discussion about that, outside of this
> > thread.
> > > My guess is that there are only a few operators that we could declare
> > > stable. Specifically, under contrib the closest one might have been
> > Kafka,
> > > but that is already superseded by the newer versions.
> > >
> > > Thomas
> > >
> > >
> > > On Fri, Jul 14, 2017 at 10:21 AM, Pramod Immaneni <
> > pramod@datatorrent.com>
> > > wrote:
> > >
> > > > We had a discussion a while back, agreed to relegate non-stable and
> > > > experimental operators to contrib and also added this to contribution
> > > > guidelines. We aexecuted on this and cleaned up the repo by moving
> > > > operators deemed non-suitable for production use at that time to
> > contrib.
> > > > So I wouldn't say the operators that are at the top level today or
> the
> > > ones
> > > > in library are 0.x.x quality. Granted, we may need to do one more
> pass
> > as
> > > > some of the operators may no longer be considered the right
> > > implementations
> > > > with the advent of the windowed operator.
> > > >
> > > > Since contrib used to be the place that housed operators that
> required
> > > > third party libraries, there are some production quality operators in
> > > there
> > > > that need to be pulled out into top level modules.
> > > >
> > > > I think we are in agreement that for operators that we consider
> stable,
> > > we
> > > > should provide a b/w stub. I would add that we strongly consider
> > creating
> > > > the org.apache.apex counterparts of any stable operators that are in
> > > > contrib out in top level modules and have the com.datatorrent stubs
> in
> > > > contrib.
> > > >
> > > > For the operators not considered stable, I would prefer we either
> leave
> > > the
> > > > current package path as is or if we change the package path then
> create
> > > the
> > > > b/w stub as I am not sure how widely they are in use (so, in essence,
> > > > preserve semantic versioning). It would be good if there is a
> followup
> > > JIRA
> > > > that takes a look at what other operators can be moved to contrib
> with
> > > the
> > > > advent of the newer frameworks and understanding.
> > > >
> > > > Thanks
> > > >
> > > > On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise <thw@apache.org>
> wrote:
> > > >
> > > > > Most of the operators evolve, as is quite clearly indicated by
> > > @Evolving
> > > > > annotations. So while perhaps 0.x.x would be a more appropriate
> > version
> > > > > number, I don't think you can change that.
> > > > >
> > > > > Thomas
> > > > >
> > > > > On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov <
> v.rozov@datatorrent.com
> > >
> > > > > wrote:
> > > > >
> > > > > > If entire library is not stable, its version should be 0.x.x
and
> > > there
> > > > > > should not be any semantic versioning enabled or implied. It
> > evolves.
> > > > If
> > > > > > some operators are stable as 3.8.x version suggests, the library
> > > should
> > > > > > follow semantic versioning and it is not OK to make a major
> change
> > > that
> > > > > > breaks semantic versioning across the entire library. It is
not a
> > > > finding
> > > > > > an excuse not to make a change. For me semantic versioning and
> > > backward
> > > > > > compatibility is more important than a proper package name.
> > > > > >
> > > > > > Thank you,
> > > > > >
> > > > > > Vlad
> > > > > >
> > > > > >
> > > > > > On 7/14/17 09:11, Thomas Weise wrote:
> > > > > >
> > > > > >> Semantic versioning makes sense when you have a stable baseline.
> > As
> > > > long
> > > > > >> as
> > > > > >> frequent fixes need to be made to address basic issues,
it makes
> > no
> > > > > sense
> > > > > >> to declare operators stable, which is why they are marked
> > evolving.
> > > > > >>
> > > > > >> Instead of finding reasons to avoid changes that should
have
> been
> > > > made a
> > > > > >> long time ago, why not discuss what a "stable operator"
is and
> > which
> > > > > ones
> > > > > >> deserve to be in that category. Those are the ones that
will get
> > the
> > > > > >> backward compatibility stub.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Thomas
> > > > > >>
> > > > > >>
> > > > > >> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <
> > > v.rozov@datatorrent.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> My preference is to agree that the next Malhar release is
4.0.0
> > and
> > > > make
> > > > > >>> the proposed changes when 3.8.0 goes out. Otherwise
why to keep
> > > > > semantic
> > > > > >>> versioning checks on Malhar in case there is no version
> > > > compatibility.
> > > > > >>>
> > > > > >>> Thank you,
> > > > > >>>
> > > > > >>> Vlad
> > > > > >>>
> > > > > >>>
> > > > > >>> On 7/14/17 08:11, Thomas Weise wrote:
> > > > > >>>
> > > > > >>> next release is 3.8.0
> > > > > >>>>
> > > > > >>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <
> > > > v.rozov@datatorrent.com>
> > > > > >>>> wrote:
> > > > > >>>>
> > > > > >>>> next release - 3.9.0 or 4.0.0?
> > > > > >>>>
> > > > > >>>>> Thank you,
> > > > > >>>>>
> > > > > >>>>> Vlad
> > > > > >>>>>
> > > > > >>>>> On 7/13/17 22:25, Thomas Weise wrote:
> > > > > >>>>>
> > > > > >>>>> It is time to resurrect this thread and get
going with the
> > work.
> > > > > >>>>>
> > > > > >>>>>> For the next release, I will sign up to
do the package move
> in
> > > > > Malhar:
> > > > > >>>>>>
> > > > > >>>>>> https://issues.apache.org/jira/browse/APEXMALHAR-2517
> > > > > >>>>>>
> > > > > >>>>>> In general this will be straightforward;
most classes in
> > Malhar
> > > > are
> > > > > >>>>>> marked
> > > > > >>>>>> evolving and it is trivial for users to
change import
> > > statements.
> > > > > >>>>>> However,
> > > > > >>>>>> I would suggest to discuss if there are
selected operators
> > that
> > > > are
> > > > > >>>>>> worth
> > > > > >>>>>> keeping a backward compatibility stub in
the old location.
> > > > > >>>>>>
> > > > > >>>>>> Here is my plan:
> > > > > >>>>>>
> > > > > >>>>>> 1. Relocate all classes in *lib* and *contrib*
within one
> PR -
> > > > this
> > > > > is
> > > > > >>>>>> all
> > > > > >>>>>>
> > > > > >>>>>> IDE automated work
> > > > > >>>>>> 2. Add backward compatibility classes, if,
any in separate
> PR
> > > > > >>>>>> 3. Create PR for Megh library to reflect
moved classes
> > > > > >>>>>>
> > > > > >>>>>> Thanks,
> > > > > >>>>>> Thomas
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni
<
> > > > > >>>>>> pramod@datatorrent.com
> > > > > >>>>>> wrote:
> > > > > >>>>>>
> > > > > >>>>>> Inline
> > > > > >>>>>>
> > > > > >>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas
Weise <
> > thw@apache.org>
> > > > > >>>>>>> wrote:
> > > > > >>>>>>>
> > > > > >>>>>>> -->
> > > > > >>>>>>>
> > > > > >>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod
Immaneni <
> > > > > >>>>>>>> pramod@datatorrent.com
> > > > > >>>>>>>>
> > > > > >>>>>>>> wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>> For malhar, for existing operators,
I prefer we do this as
> > > part
> > > > of
> > > > > >>>>>>>> the
> > > > > >>>>>>>>
> > > > > >>>>>>>> planned refactoring for breaking
the monolith modules into
> > > baby
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> packages
> > > > > >>>>>>>>>
> > > > > >>>>>>>> and would also prefer deprecating
the existing operators
> in
> > > > place.
> > > > > >>>>>>>> Refactor into smaller modules was
discussed for
> > malhar-contrib
> > > > and
> > > > > >>>>>>>> given
> > > > > >>>>>>>> the overall state of that module
I think it is OK to defer
> > > > package
> > > > > >>>>>>>>
> > > > > >>>>>>>> renaming
> > > > > >>>>>>>>
> > > > > >>>>>>> there. I do however prefer to see the
package rename
> > addressed
> > > > for
> > > > > >>>>>>>
> > > > > >>>>>>>> other
> > > > > >>>>>>>> modules, especially for the main
library module.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Should we consider breaking the
library into smaller
> modules
> > > as
> > > > > >>>>>>>> well,
> > > > > >>>>>>>>
> > > > > >>>>>>>> the
> > > > > >>>>>>> file/block operators for example probably
can be in their
> own
> > > > > module
> > > > > >>>>>>> from
> > > > > >>>>>>> just an organizational perspective.
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> This
> > > > > >>>>>>>
> > > > > >>>>>>>> will help us achieve two things.
First, the user will see
> > all
> > > > the
> > > > > >>>>>>>>> new
> > > > > >>>>>>>>> changes at once as opposed to
dealing with it twice (with
> > > > package
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> rename
> > > > > >>>>>>>>>
> > > > > >>>>>>>> and dependency changes) and second
it will allow for a
> > > smoother
> > > > > >>>>>>>> transition
> > > > > >>>>>>>>
> > > > > >>>>>>>> as the existing code will still
work in a deprecated
> state.
> > It
> > > > > will
> > > > > >>>>>>>>
> > > > > >>>>>>>>> also
> > > > > >>>>>>>>>
> > > > > >>>>>>>> give a more consistent structure
to malhar. For new
> > operators,
> > > > we
> > > > > >>>>>>>> can
> > > > > >>>>>>>> go
> > > > > >>>>>>>> with the new package path but we
need to ensure they will
> > get
> > > > > moved
> > > > > >>>>>>>> into
> > > > > >>>>>>>> the baby packages as well.
> > > > > >>>>>>>>
> > > > > >>>>>>>> I think existing operators should
be renamed so that git
> > > history
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> remains. A
> > > > > >>>>>>>>
> > > > > >>>>>>> possible solution for backward compatibility
could be to
> > > > > subsequently
> > > > > >>>>>>>
> > > > > >>>>>>>> add
> > > > > >>>>>>>> empty subclasses in the previous
location (for existing
> > > concrete
> > > > > >>>>>>>>
> > > > > >>>>>>>> operators
> > > > > >>>>>>>>
> > > > > >>>>>>> that we know are actually in use) to
simplify migration for
> > > > users.
> > > > > >>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>> Yes we can do that.
> > > > > >>>>>>>>
> > > > > >>>>>>>> For demos, we can modify the paths
as the apps are
> typically
> > > > used
> > > > > >>>>>>>
> > > > > >>>>>>> wholesale
> > > > > >>>>>>>>
> > > > > >>>>>>>> and the interface is typically manual
interaction.
> > > > > >>>>>>>>
> > > > > >>>>>>>>> For core, if we are adding new
api subsystems, like the
> > > > launcher
> > > > > >>>>>>>>> api
> > > > > >>>>>>>>> we
> > > > > >>>>>>>>> added recently for example,
we can go with new package
> path
> > > but
> > > > > if
> > > > > >>>>>>>>> we
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> are
> > > > > >>>>>>>>>
> > > > > >>>>>>>> making incremental additions to
existing functionality, I
> > feel
> > > > it
> > > > > is
> > > > > >>>>>>>> better
> > > > > >>>>>>>>
> > > > > >>>>>>>> to keep it in the same package.
I also prefer we keep the
> > > > package
> > > > > of
> > > > > >>>>>>>>
> > > > > >>>>>>>>> the
> > > > > >>>>>>>>>
> > > > > >>>>>>>> implementation classes consistent
with api, for
> > > > understandability
> > > > > >>>>>>>> and
> > > > > >>>>>>>>
> > > > > >>>>>>>> readability of the code. So, for
example, we don't change
> > > > package
> > > > > >>>>>>>>> path
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> of
> > > > > >>>>>>>>>
> > > > > >>>>>>>> LogicalPlan as it is an implementation
of DAG. It is
> > > subjective,
> > > > > but
> > > > > >>>>>>>> it
> > > > > >>>>>>>>
> > > > > >>>>>>>> will be good if we can also do the
same with classes
> closely
> > > > > related
> > > > > >>>>>>>>> to
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> the
> > > > > >>>>>>>>>
> > > > > >>>>>>>> implementation classes as well.
Maybe we can moving these
> > on a
> > > > > >>>>>>>> package
> > > > > >>>>>>>>
> > > > > >>>>>>>>> by
> > > > > >>>>>>>>>
> > > > > >>>>>>>> package basis, like everything in
> > com.datatorrent.stram.engine
> > > > > could
> > > > > >>>>>>>> be
> > > > > >>>>>>>>
> > > > > >>>>>>>> moved. For completely internal components
like buffer
> > server,
> > > we
> > > > > can
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> move
> > > > > >>>>>>>>>
> > > > > >>>>>>>> them wholesale. We can consider
moving all api and
> classes,
> > > when
> > > > > we
> > > > > >>>>>>>> go
> > > > > >>>>>>>> to
> > > > > >>>>>>>> next major release but would like
to see if we can find a
> > way
> > > to
> > > > > >>>>>>>> support
> > > > > >>>>>>>> existing api for one more major
release in deprecated
> mode.
> > > > > >>>>>>>>
> > > > > >>>>>>>> The point of the major release is
to enable backward
> > > > incompatible
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> changes
> > > > > >>>>>>>> and I don't think it is realistic
to support the existing
> > API
> > > > for
> > > > > >>>>>>>> another
> > > > > >>>>>>>> major release. IMO it is also not
necessary as most
> existing
> > > > > >>>>>>>> application
> > > > > >>>>>>>> code refers to operators, attributes
and the application
> > > > > interface.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Perhaps
> > > > > >>>>>>>>
> > > > > >>>>>>> it is possible to keep those around
as interface extensions
> > to
> > > > help
> > > > > >>>>>>>
> > > > > >>>>>>>> migration. Custom operators may
need to be migrated to
> > reflect
> > > > API
> > > > > >>>>>>>>
> > > > > >>>>>>>> changes,
> > > > > >>>>>>>>
> > > > > >>>>>>> and I would consider that a reasonable
task for operator
> > > > developers
> > > > > >>>>>>> as
> > > > > >>>>>>>
> > > > > >>>>>>>> part
> > > > > >>>>>>>>
> > > > > >>>>>>> of a major upgrade.
> > > > > >>>>>>>
> > > > > >>>>>>>> It would be good if we can keep
them as deprecated
> interface
> > > > > >>>>>>>> extensions
> > > > > >>>>>>>>
> > > > > >>>>>>>> for
> > > > > >>>>>>> one release to provide a smoother transition.
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> API and implementation in engine are
kept separate
> > > intentionally.
> > > > > >>>>>>> They
> > > > > >>>>>>>
> > > > > >>>>>>> reside in different packages today,
so I don't see a
> problem
> > > > > renaming
> > > > > >>>>>>>> com.datatorrent.stram.engine as
you say, even when the API
> > > > cannot
> > > > > be
> > > > > >>>>>>>> touched right away.
> > > > > >>>>>>>>
> > > > > >>>>>>>> They are different packages but
sharing a common prefix
> with
> > > api
> > > > > >>>>>>>> will
> > > > > >>>>>>>> be
> > > > > >>>>>>>>
> > > > > >>>>>>>> helpful to someone new to codebase
in terms of
> readability.
> > > Not
> > > > a
> > > > > >>>>>>> big
> > > > > >>>>>>> deal
> > > > > >>>>>>> and can be changed.
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> Thanks
> > > > > >>>>>>>
> > > > > >>>>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas
Weise <
> > thw@apache.org>
> > > > > >>>>>>>> wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>>> Hi,
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> This topic has come up on several
PRs and I think it
> > > warrants a
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> broader
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>> discussion.
> > > > > >>>>>>>>
> > > > > >>>>>>>> At the time of incubation, the decision
was to defer
> change
> > of
> > > > > Java
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> packages from com.datatorrent
to org.apache.apex till
> next
> > > > major
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> release
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>> to
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> ensure backward compatibility
for users.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> Unfortunately that has lead
to some confusion, as
> > > contributors
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> continue
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>> to
> > > > > >>>>>>>>
> > > > > >>>>>>>> add new code under legacy packages.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> It is also a wider issue
that examples for using Apex
> > > continue
> > > > > to
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> refer
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>> to
> > > > > >>>>>>>>
> > > > > >>>>>>>> com.datatorrent packages, nearly
one year after
> graduation.
> > > More
> > > > > and
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> more
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>> user code is being built on
top of something that needs
> to
> > > > > change,
> > > > > >>>>>>>>> the
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> can
> > > > > >>>>>>>>
> > > > > >>>>>>>> is being kicked down the road and
users will face more
> > changes
> > > > > >>>>>>>>> later.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> I would like to propose
the following:
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> 1. All new code has to be
submitted under
> org.apache.apex
> > > > > packages
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> 2. Not all code is under
backward compatibility
> > restriction
> > > > and
> > > > > in
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> those
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>> cases we can rename the packages
right away. Examples:
> > buffer
> > > > > >>>>>>>>> server,
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> engine, demos/examples, benchmarks
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> 3. Discuss when the core
API and operators can be
> changed.
> > > For
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> operators
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>> we
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> have a bit more freedom to do
changes before a major
> > release
> > > as
> > > > > >>>>>>>>> most
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> of
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>> them are marked @Evolving and
users have the ability to
> > > > continue
> > > > > >>>>>>>>
> > > > > >>>>>>>> using
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> prior version of Malhar with
newer engine due to engine
> > > > backward
> > > > > >>>>>>>>
> > > > > >>>>>>>> compatibility guarantee.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> Thanks,
> > > > > >>>>>>>>>> Thomas
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message