apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pramod Immaneni <pra...@datatorrent.com>
Subject Re: Java packages: legacy -> org.apache.apex
Date Fri, 14 Jul 2017 17:21:13 GMT
We had a discussion a while back, agreed to relegate non-stable and
experimental operators to contrib and also added this to contribution
guidelines. We aexecuted on this and cleaned up the repo by moving
operators deemed non-suitable for production use at that time to contrib.
So I wouldn't say the operators that are at the top level today or the ones
in library are 0.x.x quality. Granted, we may need to do one more pass as
some of the operators may no longer be considered the right implementations
with the advent of the windowed operator.

Since contrib used to be the place that housed operators that required
third party libraries, there are some production quality operators in there
that need to be pulled out into top level modules.

I think we are in agreement that for operators that we consider stable, we
should provide a b/w stub. I would add that we strongly consider creating
the org.apache.apex counterparts of any stable operators that are in
contrib out in top level modules and have the com.datatorrent stubs in
contrib.

For the operators not considered stable, I would prefer we either leave the
current package path as is or if we change the package path then create the
b/w stub as I am not sure how widely they are in use (so, in essence,
preserve semantic versioning). It would be good if there is a followup JIRA
that takes a look at what other operators can be moved to contrib with the
advent of the newer frameworks and understanding.

Thanks

On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise <thw@apache.org> wrote:

> Most of the operators evolve, as is quite clearly indicated by @Evolving
> annotations. So while perhaps 0.x.x would be a more appropriate version
> number, I don't think you can change that.
>
> Thomas
>
> On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov <v.rozov@datatorrent.com>
> wrote:
>
> > If entire library is not stable, its version should be 0.x.x and there
> > should not be any semantic versioning enabled or implied. It evolves. If
> > some operators are stable as 3.8.x version suggests, the library should
> > follow semantic versioning and it is not OK to make a major change that
> > breaks semantic versioning across the entire library. It is not a finding
> > an excuse not to make a change. For me semantic versioning and backward
> > compatibility is more important than a proper package name.
> >
> > Thank you,
> >
> > Vlad
> >
> >
> > On 7/14/17 09:11, Thomas Weise wrote:
> >
> >> Semantic versioning makes sense when you have a stable baseline. As long
> >> as
> >> frequent fixes need to be made to address basic issues, it makes no
> sense
> >> to declare operators stable, which is why they are marked evolving.
> >>
> >> Instead of finding reasons to avoid changes that should have been made a
> >> long time ago, why not discuss what a "stable operator" is and which
> ones
> >> deserve to be in that category. Those are the ones that will get the
> >> backward compatibility stub.
> >>
> >> Thanks,
> >> Thomas
> >>
> >>
> >> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <v.rozov@datatorrent.com>
> >> wrote:
> >>
> >> My preference is to agree that the next Malhar release is 4.0.0 and make
> >>> the proposed changes when 3.8.0 goes out. Otherwise why to keep
> semantic
> >>> versioning checks on Malhar in case there is no version compatibility.
> >>>
> >>> Thank you,
> >>>
> >>> Vlad
> >>>
> >>>
> >>> On 7/14/17 08:11, Thomas Weise wrote:
> >>>
> >>> next release is 3.8.0
> >>>>
> >>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <v.rozov@datatorrent.com>
> >>>> wrote:
> >>>>
> >>>> next release - 3.9.0 or 4.0.0?
> >>>>
> >>>>> Thank you,
> >>>>>
> >>>>> Vlad
> >>>>>
> >>>>> On 7/13/17 22:25, Thomas Weise wrote:
> >>>>>
> >>>>> It is time to resurrect this thread and get going with the work.
> >>>>>
> >>>>>> For the next release, I will sign up to do the package move
in
> Malhar:
> >>>>>>
> >>>>>> https://issues.apache.org/jira/browse/APEXMALHAR-2517
> >>>>>>
> >>>>>> In general this will be straightforward; most classes in Malhar
are
> >>>>>> marked
> >>>>>> evolving and it is trivial for users to change import statements.
> >>>>>> However,
> >>>>>> I would suggest to discuss if there are selected operators that
are
> >>>>>> worth
> >>>>>> keeping a backward compatibility stub in the old location.
> >>>>>>
> >>>>>> Here is my plan:
> >>>>>>
> >>>>>> 1. Relocate all classes in *lib* and *contrib* within one PR
- this
> is
> >>>>>> all
> >>>>>>
> >>>>>> IDE automated work
> >>>>>> 2. Add backward compatibility classes, if, any in separate PR
> >>>>>> 3. Create PR for Megh library to reflect moved classes
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Thomas
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <
> >>>>>> pramod@datatorrent.com
> >>>>>> wrote:
> >>>>>>
> >>>>>> Inline
> >>>>>>
> >>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <thw@apache.org>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>> -->
> >>>>>>>
> >>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
> >>>>>>>> pramod@datatorrent.com
> >>>>>>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> For malhar, for existing operators, I prefer we do this
as part of
> >>>>>>>> the
> >>>>>>>>
> >>>>>>>> planned refactoring for breaking the monolith modules
into baby
> >>>>>>>>>
> >>>>>>>>> packages
> >>>>>>>>>
> >>>>>>>> and would also prefer deprecating the existing operators
in place.
> >>>>>>>> Refactor into smaller modules was discussed for malhar-contrib
and
> >>>>>>>> given
> >>>>>>>> the overall state of that module I think it is OK to
defer package
> >>>>>>>>
> >>>>>>>> renaming
> >>>>>>>>
> >>>>>>> there. I do however prefer to see the package rename addressed
for
> >>>>>>>
> >>>>>>>> other
> >>>>>>>> modules, especially for the main library module.
> >>>>>>>>
> >>>>>>>> Should we consider breaking the library into smaller
modules as
> >>>>>>>> well,
> >>>>>>>>
> >>>>>>>> the
> >>>>>>> file/block operators for example probably can be in their
own
> module
> >>>>>>> from
> >>>>>>> just an organizational perspective.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> This
> >>>>>>>
> >>>>>>>> will help us achieve two things. First, the user will
see all the
> >>>>>>>>> new
> >>>>>>>>> changes at once as opposed to dealing with it twice
(with package
> >>>>>>>>>
> >>>>>>>>> rename
> >>>>>>>>>
> >>>>>>>> and dependency changes) and second it will allow for
a smoother
> >>>>>>>> transition
> >>>>>>>>
> >>>>>>>> as the existing code will still work in a deprecated
state. It
> will
> >>>>>>>>
> >>>>>>>>> also
> >>>>>>>>>
> >>>>>>>> give a more consistent structure to malhar. For new
operators, we
> >>>>>>>> can
> >>>>>>>> go
> >>>>>>>> with the new package path but we need to ensure they
will get
> moved
> >>>>>>>> into
> >>>>>>>> the baby packages as well.
> >>>>>>>>
> >>>>>>>> I think existing operators should be renamed so that
git history
> >>>>>>>>>
> >>>>>>>>> remains. A
> >>>>>>>>
> >>>>>>> possible solution for backward compatibility could be to
> subsequently
> >>>>>>>
> >>>>>>>> add
> >>>>>>>> empty subclasses in the previous location (for existing
concrete
> >>>>>>>>
> >>>>>>>> operators
> >>>>>>>>
> >>>>>>> that we know are actually in use) to simplify migration
for users.
> >>>>>>>
> >>>>>>>>
> >>>>>>>> Yes we can do that.
> >>>>>>>>
> >>>>>>>> For demos, we can modify the paths as the apps are typically
used
> >>>>>>>
> >>>>>>> wholesale
> >>>>>>>>
> >>>>>>>> and the interface is typically manual interaction.
> >>>>>>>>
> >>>>>>>>> For core, if we are adding new api subsystems, like
the launcher
> >>>>>>>>> api
> >>>>>>>>> we
> >>>>>>>>> added recently for example, we can go with new package
path but
> if
> >>>>>>>>> we
> >>>>>>>>>
> >>>>>>>>> are
> >>>>>>>>>
> >>>>>>>> making incremental additions to existing functionality,
I feel it
> is
> >>>>>>>> better
> >>>>>>>>
> >>>>>>>> to keep it in the same package. I also prefer we keep
the package
> of
> >>>>>>>>
> >>>>>>>>> the
> >>>>>>>>>
> >>>>>>>> implementation classes consistent with api, for understandability
> >>>>>>>> and
> >>>>>>>>
> >>>>>>>> readability of the code. So, for example, we don't change
package
> >>>>>>>>> path
> >>>>>>>>>
> >>>>>>>>> of
> >>>>>>>>>
> >>>>>>>> LogicalPlan as it is an implementation of DAG. It is
subjective,
> but
> >>>>>>>> it
> >>>>>>>>
> >>>>>>>> will be good if we can also do the same with classes
closely
> related
> >>>>>>>>> to
> >>>>>>>>>
> >>>>>>>>> the
> >>>>>>>>>
> >>>>>>>> implementation classes as well. Maybe we can moving
these on a
> >>>>>>>> package
> >>>>>>>>
> >>>>>>>>> by
> >>>>>>>>>
> >>>>>>>> package basis, like everything in com.datatorrent.stram.engine
> could
> >>>>>>>> be
> >>>>>>>>
> >>>>>>>> moved. For completely internal components like buffer
server, we
> can
> >>>>>>>>>
> >>>>>>>>> move
> >>>>>>>>>
> >>>>>>>> them wholesale. We can consider moving all api and classes,
when
> we
> >>>>>>>> go
> >>>>>>>> to
> >>>>>>>> next major release but would like to see if we can find
a way to
> >>>>>>>> support
> >>>>>>>> existing api for one more major release in deprecated
mode.
> >>>>>>>>
> >>>>>>>> The point of the major release is to enable backward
incompatible
> >>>>>>>>>
> >>>>>>>>> changes
> >>>>>>>> and I don't think it is realistic to support the existing
API for
> >>>>>>>> another
> >>>>>>>> major release. IMO it is also not necessary as most
existing
> >>>>>>>> application
> >>>>>>>> code refers to operators, attributes and the application
> interface.
> >>>>>>>>
> >>>>>>>> Perhaps
> >>>>>>>>
> >>>>>>> it is possible to keep those around as interface extensions
to help
> >>>>>>>
> >>>>>>>> migration. Custom operators may need to be migrated
to reflect API
> >>>>>>>>
> >>>>>>>> changes,
> >>>>>>>>
> >>>>>>> and I would consider that a reasonable task for operator
developers
> >>>>>>> as
> >>>>>>>
> >>>>>>>> part
> >>>>>>>>
> >>>>>>> of a major upgrade.
> >>>>>>>
> >>>>>>>> It would be good if we can keep them as deprecated interface
> >>>>>>>> extensions
> >>>>>>>>
> >>>>>>>> for
> >>>>>>> one release to provide a smoother transition.
> >>>>>>>
> >>>>>>>
> >>>>>>> API and implementation in engine are kept separate intentionally.
> >>>>>>> They
> >>>>>>>
> >>>>>>> reside in different packages today, so I don't see a problem
> renaming
> >>>>>>>> com.datatorrent.stram.engine as you say, even when the
API cannot
> be
> >>>>>>>> touched right away.
> >>>>>>>>
> >>>>>>>> They are different packages but sharing a common prefix
with api
> >>>>>>>> will
> >>>>>>>> be
> >>>>>>>>
> >>>>>>>> helpful to someone new to codebase in terms of readability.
Not a
> >>>>>>> big
> >>>>>>> deal
> >>>>>>> and can be changed.
> >>>>>>>
> >>>>>>>
> >>>>>>> Thanks
> >>>>>>>
> >>>>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise <thw@apache.org>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi,
> >>>>>>>>>
> >>>>>>>>> This topic has come up on several PRs and I think
it warrants a
> >>>>>>>>>>
> >>>>>>>>>> broader
> >>>>>>>>>>
> >>>>>>>>> discussion.
> >>>>>>>>
> >>>>>>>> At the time of incubation, the decision was to defer
change of
> Java
> >>>>>>>>>
> >>>>>>>>>> packages from com.datatorrent to org.apache.apex
till next major
> >>>>>>>>>>
> >>>>>>>>>> release
> >>>>>>>>>>
> >>>>>>>>> to
> >>>>>>>>>
> >>>>>>>>> ensure backward compatibility for users.
> >>>>>>>>>
> >>>>>>>>>> Unfortunately that has lead to some confusion,
as contributors
> >>>>>>>>>>
> >>>>>>>>>> continue
> >>>>>>>>>>
> >>>>>>>>> to
> >>>>>>>>
> >>>>>>>> add new code under legacy packages.
> >>>>>>>>>
> >>>>>>>>>> It is also a wider issue that examples for using
Apex continue
> to
> >>>>>>>>>>
> >>>>>>>>>> refer
> >>>>>>>>>>
> >>>>>>>>> to
> >>>>>>>>
> >>>>>>>> com.datatorrent packages, nearly one year after graduation.
More
> and
> >>>>>>>>>
> >>>>>>>>>> more
> >>>>>>>>>>
> >>>>>>>>> user code is being built on top of something that
needs to
> change,
> >>>>>>>>> the
> >>>>>>>>>
> >>>>>>>>> can
> >>>>>>>>
> >>>>>>>> is being kicked down the road and users will face more
changes
> >>>>>>>>> later.
> >>>>>>>>>
> >>>>>>>>>> I would like to propose the following:
> >>>>>>>>>>
> >>>>>>>>>> 1. All new code has to be submitted under org.apache.apex
> packages
> >>>>>>>>>>
> >>>>>>>>>> 2. Not all code is under backward compatibility
restriction and
> in
> >>>>>>>>>>
> >>>>>>>>>> those
> >>>>>>>>>>
> >>>>>>>>> cases we can rename the packages right away. Examples:
buffer
> >>>>>>>>> server,
> >>>>>>>>>
> >>>>>>>>> engine, demos/examples, benchmarks
> >>>>>>>>>>
> >>>>>>>>>> 3. Discuss when the core API and operators can
be changed. For
> >>>>>>>>>>
> >>>>>>>>>> operators
> >>>>>>>>>>
> >>>>>>>>> we
> >>>>>>>>>
> >>>>>>>>> have a bit more freedom to do changes before a major
release as
> >>>>>>>>> most
> >>>>>>>>>
> >>>>>>>>>> of
> >>>>>>>>>>
> >>>>>>>>> them are marked @Evolving and users have the ability
to continue
> >>>>>>>>
> >>>>>>>> using
> >>>>>>>>>
> >>>>>>>>> prior version of Malhar with newer engine due to
engine backward
> >>>>>>>>
> >>>>>>>> compatibility guarantee.
> >>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> Thomas
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message