apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pramod Immaneni <pra...@datatorrent.com>
Subject Re: Java packages: legacy -> org.apache.apex
Date Fri, 14 Jul 2017 18:33:28 GMT
How about coming up with a list of @Evolving operators that we would like
to see in the eventual stable list and move those along with the not
@Evolving ones in org.apache.apex with b/w stubs and leave the rest as they
are. Then have a follow up JIRA for the rest to be moved over to contrib
and be deprecated.

Thanks

On Fri, Jul 14, 2017 at 10:37 AM, Thomas Weise <thomas.weise@gmail.com>
wrote:

> We need to keep the discussion here on topic, if other things are piled on
> top then nothing gets done.
>
> Most operators today are not stable, they are @Evolving. So perhaps it is
> necessary to have a separate discussion about that, outside of this thread.
> My guess is that there are only a few operators that we could declare
> stable. Specifically, under contrib the closest one might have been Kafka,
> but that is already superseded by the newer versions.
>
> Thomas
>
>
> On Fri, Jul 14, 2017 at 10:21 AM, Pramod Immaneni <pramod@datatorrent.com>
> wrote:
>
> > We had a discussion a while back, agreed to relegate non-stable and
> > experimental operators to contrib and also added this to contribution
> > guidelines. We aexecuted on this and cleaned up the repo by moving
> > operators deemed non-suitable for production use at that time to contrib.
> > So I wouldn't say the operators that are at the top level today or the
> ones
> > in library are 0.x.x quality. Granted, we may need to do one more pass as
> > some of the operators may no longer be considered the right
> implementations
> > with the advent of the windowed operator.
> >
> > Since contrib used to be the place that housed operators that required
> > third party libraries, there are some production quality operators in
> there
> > that need to be pulled out into top level modules.
> >
> > I think we are in agreement that for operators that we consider stable,
> we
> > should provide a b/w stub. I would add that we strongly consider creating
> > the org.apache.apex counterparts of any stable operators that are in
> > contrib out in top level modules and have the com.datatorrent stubs in
> > contrib.
> >
> > For the operators not considered stable, I would prefer we either leave
> the
> > current package path as is or if we change the package path then create
> the
> > b/w stub as I am not sure how widely they are in use (so, in essence,
> > preserve semantic versioning). It would be good if there is a followup
> JIRA
> > that takes a look at what other operators can be moved to contrib with
> the
> > advent of the newer frameworks and understanding.
> >
> > Thanks
> >
> > On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise <thw@apache.org> wrote:
> >
> > > Most of the operators evolve, as is quite clearly indicated by
> @Evolving
> > > annotations. So while perhaps 0.x.x would be a more appropriate version
> > > number, I don't think you can change that.
> > >
> > > Thomas
> > >
> > > On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov <v.rozov@datatorrent.com>
> > > wrote:
> > >
> > > > If entire library is not stable, its version should be 0.x.x and
> there
> > > > should not be any semantic versioning enabled or implied. It evolves.
> > If
> > > > some operators are stable as 3.8.x version suggests, the library
> should
> > > > follow semantic versioning and it is not OK to make a major change
> that
> > > > breaks semantic versioning across the entire library. It is not a
> > finding
> > > > an excuse not to make a change. For me semantic versioning and
> backward
> > > > compatibility is more important than a proper package name.
> > > >
> > > > Thank you,
> > > >
> > > > Vlad
> > > >
> > > >
> > > > On 7/14/17 09:11, Thomas Weise wrote:
> > > >
> > > >> Semantic versioning makes sense when you have a stable baseline. As
> > long
> > > >> as
> > > >> frequent fixes need to be made to address basic issues, it makes no
> > > sense
> > > >> to declare operators stable, which is why they are marked evolving.
> > > >>
> > > >> Instead of finding reasons to avoid changes that should have been
> > made a
> > > >> long time ago, why not discuss what a "stable operator" is and which
> > > ones
> > > >> deserve to be in that category. Those are the ones that will get the
> > > >> backward compatibility stub.
> > > >>
> > > >> Thanks,
> > > >> Thomas
> > > >>
> > > >>
> > > >> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <
> v.rozov@datatorrent.com>
> > > >> wrote:
> > > >>
> > > >> My preference is to agree that the next Malhar release is 4.0.0 and
> > make
> > > >>> the proposed changes when 3.8.0 goes out. Otherwise why to keep
> > > semantic
> > > >>> versioning checks on Malhar in case there is no version
> > compatibility.
> > > >>>
> > > >>> Thank you,
> > > >>>
> > > >>> Vlad
> > > >>>
> > > >>>
> > > >>> On 7/14/17 08:11, Thomas Weise wrote:
> > > >>>
> > > >>> next release is 3.8.0
> > > >>>>
> > > >>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <
> > v.rozov@datatorrent.com>
> > > >>>> wrote:
> > > >>>>
> > > >>>> next release - 3.9.0 or 4.0.0?
> > > >>>>
> > > >>>>> Thank you,
> > > >>>>>
> > > >>>>> Vlad
> > > >>>>>
> > > >>>>> On 7/13/17 22:25, Thomas Weise wrote:
> > > >>>>>
> > > >>>>> It is time to resurrect this thread and get going with
the work.
> > > >>>>>
> > > >>>>>> For the next release, I will sign up to do the package
move in
> > > Malhar:
> > > >>>>>>
> > > >>>>>> https://issues.apache.org/jira/browse/APEXMALHAR-2517
> > > >>>>>>
> > > >>>>>> In general this will be straightforward; most classes
in Malhar
> > are
> > > >>>>>> marked
> > > >>>>>> evolving and it is trivial for users to change import
> statements.
> > > >>>>>> However,
> > > >>>>>> I would suggest to discuss if there are selected operators
that
> > are
> > > >>>>>> worth
> > > >>>>>> keeping a backward compatibility stub in the old location.
> > > >>>>>>
> > > >>>>>> Here is my plan:
> > > >>>>>>
> > > >>>>>> 1. Relocate all classes in *lib* and *contrib* within
one PR -
> > this
> > > is
> > > >>>>>> all
> > > >>>>>>
> > > >>>>>> IDE automated work
> > > >>>>>> 2. Add backward compatibility classes, if, any in
separate PR
> > > >>>>>> 3. Create PR for Megh library to reflect moved classes
> > > >>>>>>
> > > >>>>>> Thanks,
> > > >>>>>> Thomas
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <
> > > >>>>>> pramod@datatorrent.com
> > > >>>>>> wrote:
> > > >>>>>>
> > > >>>>>> Inline
> > > >>>>>>
> > > >>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <thw@apache.org>
> > > >>>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>> -->
> > > >>>>>>>
> > > >>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni
<
> > > >>>>>>>> pramod@datatorrent.com
> > > >>>>>>>>
> > > >>>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>> For malhar, for existing operators, I prefer
we do this as
> part
> > of
> > > >>>>>>>> the
> > > >>>>>>>>
> > > >>>>>>>> planned refactoring for breaking the monolith
modules into
> baby
> > > >>>>>>>>>
> > > >>>>>>>>> packages
> > > >>>>>>>>>
> > > >>>>>>>> and would also prefer deprecating the existing
operators in
> > place.
> > > >>>>>>>> Refactor into smaller modules was discussed
for malhar-contrib
> > and
> > > >>>>>>>> given
> > > >>>>>>>> the overall state of that module I think it
is OK to defer
> > package
> > > >>>>>>>>
> > > >>>>>>>> renaming
> > > >>>>>>>>
> > > >>>>>>> there. I do however prefer to see the package
rename addressed
> > for
> > > >>>>>>>
> > > >>>>>>>> other
> > > >>>>>>>> modules, especially for the main library module.
> > > >>>>>>>>
> > > >>>>>>>> Should we consider breaking the library into
smaller modules
> as
> > > >>>>>>>> well,
> > > >>>>>>>>
> > > >>>>>>>> the
> > > >>>>>>> file/block operators for example probably can
be in their own
> > > module
> > > >>>>>>> from
> > > >>>>>>> just an organizational perspective.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> This
> > > >>>>>>>
> > > >>>>>>>> will help us achieve two things. First, the
user will see all
> > the
> > > >>>>>>>>> new
> > > >>>>>>>>> changes at once as opposed to dealing
with it twice (with
> > package
> > > >>>>>>>>>
> > > >>>>>>>>> rename
> > > >>>>>>>>>
> > > >>>>>>>> and dependency changes) and second it will
allow for a
> smoother
> > > >>>>>>>> transition
> > > >>>>>>>>
> > > >>>>>>>> as the existing code will still work in a
deprecated state. It
> > > will
> > > >>>>>>>>
> > > >>>>>>>>> also
> > > >>>>>>>>>
> > > >>>>>>>> give a more consistent structure to malhar.
For new operators,
> > we
> > > >>>>>>>> can
> > > >>>>>>>> go
> > > >>>>>>>> with the new package path but we need to ensure
they will get
> > > moved
> > > >>>>>>>> into
> > > >>>>>>>> the baby packages as well.
> > > >>>>>>>>
> > > >>>>>>>> I think existing operators should be renamed
so that git
> history
> > > >>>>>>>>>
> > > >>>>>>>>> remains. A
> > > >>>>>>>>
> > > >>>>>>> possible solution for backward compatibility could
be to
> > > subsequently
> > > >>>>>>>
> > > >>>>>>>> add
> > > >>>>>>>> empty subclasses in the previous location
(for existing
> concrete
> > > >>>>>>>>
> > > >>>>>>>> operators
> > > >>>>>>>>
> > > >>>>>>> that we know are actually in use) to simplify
migration for
> > users.
> > > >>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> Yes we can do that.
> > > >>>>>>>>
> > > >>>>>>>> For demos, we can modify the paths as the
apps are typically
> > used
> > > >>>>>>>
> > > >>>>>>> wholesale
> > > >>>>>>>>
> > > >>>>>>>> and the interface is typically manual interaction.
> > > >>>>>>>>
> > > >>>>>>>>> For core, if we are adding new api subsystems,
like the
> > launcher
> > > >>>>>>>>> api
> > > >>>>>>>>> we
> > > >>>>>>>>> added recently for example, we can go
with new package path
> but
> > > if
> > > >>>>>>>>> we
> > > >>>>>>>>>
> > > >>>>>>>>> are
> > > >>>>>>>>>
> > > >>>>>>>> making incremental additions to existing functionality,
I feel
> > it
> > > is
> > > >>>>>>>> better
> > > >>>>>>>>
> > > >>>>>>>> to keep it in the same package. I also prefer
we keep the
> > package
> > > of
> > > >>>>>>>>
> > > >>>>>>>>> the
> > > >>>>>>>>>
> > > >>>>>>>> implementation classes consistent with api,
for
> > understandability
> > > >>>>>>>> and
> > > >>>>>>>>
> > > >>>>>>>> readability of the code. So, for example,
we don't change
> > package
> > > >>>>>>>>> path
> > > >>>>>>>>>
> > > >>>>>>>>> of
> > > >>>>>>>>>
> > > >>>>>>>> LogicalPlan as it is an implementation of
DAG. It is
> subjective,
> > > but
> > > >>>>>>>> it
> > > >>>>>>>>
> > > >>>>>>>> will be good if we can also do the same with
classes closely
> > > related
> > > >>>>>>>>> to
> > > >>>>>>>>>
> > > >>>>>>>>> the
> > > >>>>>>>>>
> > > >>>>>>>> implementation classes as well. Maybe we can
moving these on a
> > > >>>>>>>> package
> > > >>>>>>>>
> > > >>>>>>>>> by
> > > >>>>>>>>>
> > > >>>>>>>> package basis, like everything in com.datatorrent.stram.engine
> > > could
> > > >>>>>>>> be
> > > >>>>>>>>
> > > >>>>>>>> moved. For completely internal components
like buffer server,
> we
> > > can
> > > >>>>>>>>>
> > > >>>>>>>>> move
> > > >>>>>>>>>
> > > >>>>>>>> them wholesale. We can consider moving all
api and classes,
> when
> > > we
> > > >>>>>>>> go
> > > >>>>>>>> to
> > > >>>>>>>> next major release but would like to see if
we can find a way
> to
> > > >>>>>>>> support
> > > >>>>>>>> existing api for one more major release in
deprecated mode.
> > > >>>>>>>>
> > > >>>>>>>> The point of the major release is to enable
backward
> > incompatible
> > > >>>>>>>>>
> > > >>>>>>>>> changes
> > > >>>>>>>> and I don't think it is realistic to support
the existing API
> > for
> > > >>>>>>>> another
> > > >>>>>>>> major release. IMO it is also not necessary
as most existing
> > > >>>>>>>> application
> > > >>>>>>>> code refers to operators, attributes and the
application
> > > interface.
> > > >>>>>>>>
> > > >>>>>>>> Perhaps
> > > >>>>>>>>
> > > >>>>>>> it is possible to keep those around as interface
extensions to
> > help
> > > >>>>>>>
> > > >>>>>>>> migration. Custom operators may need to be
migrated to reflect
> > API
> > > >>>>>>>>
> > > >>>>>>>> changes,
> > > >>>>>>>>
> > > >>>>>>> and I would consider that a reasonable task for
operator
> > developers
> > > >>>>>>> as
> > > >>>>>>>
> > > >>>>>>>> part
> > > >>>>>>>>
> > > >>>>>>> of a major upgrade.
> > > >>>>>>>
> > > >>>>>>>> It would be good if we can keep them as deprecated
interface
> > > >>>>>>>> extensions
> > > >>>>>>>>
> > > >>>>>>>> for
> > > >>>>>>> one release to provide a smoother transition.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> API and implementation in engine are kept separate
> intentionally.
> > > >>>>>>> They
> > > >>>>>>>
> > > >>>>>>> reside in different packages today, so I don't
see a problem
> > > renaming
> > > >>>>>>>> com.datatorrent.stram.engine as you say, even
when the API
> > cannot
> > > be
> > > >>>>>>>> touched right away.
> > > >>>>>>>>
> > > >>>>>>>> They are different packages but sharing a
common prefix with
> api
> > > >>>>>>>> will
> > > >>>>>>>> be
> > > >>>>>>>>
> > > >>>>>>>> helpful to someone new to codebase in terms
of readability.
> Not
> > a
> > > >>>>>>> big
> > > >>>>>>> deal
> > > >>>>>>> and can be changed.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> Thanks
> > > >>>>>>>
> > > >>>>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise
<thw@apache.org>
> > > >>>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Hi,
> > > >>>>>>>>>
> > > >>>>>>>>> This topic has come up on several PRs
and I think it
> warrants a
> > > >>>>>>>>>>
> > > >>>>>>>>>> broader
> > > >>>>>>>>>>
> > > >>>>>>>>> discussion.
> > > >>>>>>>>
> > > >>>>>>>> At the time of incubation, the decision was
to defer change of
> > > Java
> > > >>>>>>>>>
> > > >>>>>>>>>> packages from com.datatorrent to org.apache.apex
till next
> > major
> > > >>>>>>>>>>
> > > >>>>>>>>>> release
> > > >>>>>>>>>>
> > > >>>>>>>>> to
> > > >>>>>>>>>
> > > >>>>>>>>> ensure backward compatibility for users.
> > > >>>>>>>>>
> > > >>>>>>>>>> Unfortunately that has lead to some
confusion, as
> contributors
> > > >>>>>>>>>>
> > > >>>>>>>>>> continue
> > > >>>>>>>>>>
> > > >>>>>>>>> to
> > > >>>>>>>>
> > > >>>>>>>> add new code under legacy packages.
> > > >>>>>>>>>
> > > >>>>>>>>>> It is also a wider issue that examples
for using Apex
> continue
> > > to
> > > >>>>>>>>>>
> > > >>>>>>>>>> refer
> > > >>>>>>>>>>
> > > >>>>>>>>> to
> > > >>>>>>>>
> > > >>>>>>>> com.datatorrent packages, nearly one year
after graduation.
> More
> > > and
> > > >>>>>>>>>
> > > >>>>>>>>>> more
> > > >>>>>>>>>>
> > > >>>>>>>>> user code is being built on top of something
that needs to
> > > change,
> > > >>>>>>>>> the
> > > >>>>>>>>>
> > > >>>>>>>>> can
> > > >>>>>>>>
> > > >>>>>>>> is being kicked down the road and users will
face more changes
> > > >>>>>>>>> later.
> > > >>>>>>>>>
> > > >>>>>>>>>> I would like to propose the following:
> > > >>>>>>>>>>
> > > >>>>>>>>>> 1. All new code has to be submitted
under org.apache.apex
> > > packages
> > > >>>>>>>>>>
> > > >>>>>>>>>> 2. Not all code is under backward
compatibility restriction
> > and
> > > in
> > > >>>>>>>>>>
> > > >>>>>>>>>> those
> > > >>>>>>>>>>
> > > >>>>>>>>> cases we can rename the packages right
away. Examples: buffer
> > > >>>>>>>>> server,
> > > >>>>>>>>>
> > > >>>>>>>>> engine, demos/examples, benchmarks
> > > >>>>>>>>>>
> > > >>>>>>>>>> 3. Discuss when the core API and operators
can be changed.
> For
> > > >>>>>>>>>>
> > > >>>>>>>>>> operators
> > > >>>>>>>>>>
> > > >>>>>>>>> we
> > > >>>>>>>>>
> > > >>>>>>>>> have a bit more freedom to do changes
before a major release
> as
> > > >>>>>>>>> most
> > > >>>>>>>>>
> > > >>>>>>>>>> of
> > > >>>>>>>>>>
> > > >>>>>>>>> them are marked @Evolving and users have
the ability to
> > continue
> > > >>>>>>>>
> > > >>>>>>>> using
> > > >>>>>>>>>
> > > >>>>>>>>> prior version of Malhar with newer engine
due to engine
> > backward
> > > >>>>>>>>
> > > >>>>>>>> compatibility guarantee.
> > > >>>>>>>>>
> > > >>>>>>>>>> Thanks,
> > > >>>>>>>>>> Thomas
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message