apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Weise <...@apache.org>
Subject Re: Java packages: legacy -> org.apache.apex
Date Fri, 14 Jul 2017 16:11:11 GMT
Semantic versioning makes sense when you have a stable baseline. As long as
frequent fixes need to be made to address basic issues, it makes no sense
to declare operators stable, which is why they are marked evolving.

Instead of finding reasons to avoid changes that should have been made a
long time ago, why not discuss what a "stable operator" is and which ones
deserve to be in that category. Those are the ones that will get the
backward compatibility stub.

Thanks,
Thomas


On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <v.rozov@datatorrent.com> wrote:

> My preference is to agree that the next Malhar release is 4.0.0 and make
> the proposed changes when 3.8.0 goes out. Otherwise why to keep semantic
> versioning checks on Malhar in case there is no version compatibility.
>
> Thank you,
>
> Vlad
>
>
> On 7/14/17 08:11, Thomas Weise wrote:
>
>> next release is 3.8.0
>>
>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <v.rozov@datatorrent.com>
>> wrote:
>>
>> next release - 3.9.0 or 4.0.0?
>>>
>>> Thank you,
>>>
>>> Vlad
>>>
>>> On 7/13/17 22:25, Thomas Weise wrote:
>>>
>>> It is time to resurrect this thread and get going with the work.
>>>>
>>>> For the next release, I will sign up to do the package move in Malhar:
>>>>
>>>> https://issues.apache.org/jira/browse/APEXMALHAR-2517
>>>>
>>>> In general this will be straightforward; most classes in Malhar are
>>>> marked
>>>> evolving and it is trivial for users to change import statements.
>>>> However,
>>>> I would suggest to discuss if there are selected operators that are
>>>> worth
>>>> keeping a backward compatibility stub in the old location.
>>>>
>>>> Here is my plan:
>>>>
>>>> 1. Relocate all classes in *lib* and *contrib* within one PR - this is
>>>> all
>>>>
>>>> IDE automated work
>>>> 2. Add backward compatibility classes, if, any in separate PR
>>>> 3. Create PR for Megh library to reflect moved classes
>>>>
>>>> Thanks,
>>>> Thomas
>>>>
>>>>
>>>>
>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <pramod@datatorrent.com
>>>> >
>>>> wrote:
>>>>
>>>> Inline
>>>>
>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <thw@apache.org>
wrote:
>>>>>
>>>>> -->
>>>>>
>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
>>>>>> pramod@datatorrent.com
>>>>>>
>>>>>> wrote:
>>>>>>
>>>>>> For malhar, for existing operators, I prefer we do this as part of
the
>>>>>>
>>>>>>> planned refactoring for breaking the monolith modules into baby
>>>>>>>
>>>>>>> packages
>>>>>> and would also prefer deprecating the existing operators in place.
>>>>>> Refactor into smaller modules was discussed for malhar-contrib and
>>>>>> given
>>>>>> the overall state of that module I think it is OK to defer package
>>>>>>
>>>>>> renaming
>>>>>
>>>>> there. I do however prefer to see the package rename addressed for
>>>>>> other
>>>>>> modules, especially for the main library module.
>>>>>>
>>>>>> Should we consider breaking the library into smaller modules as well,
>>>>>>
>>>>> the
>>>>> file/block operators for example probably can be in their own module
>>>>> from
>>>>> just an organizational perspective.
>>>>>
>>>>>
>>>>>
>>>>> This
>>>>>>
>>>>>>> will help us achieve two things. First, the user will see all
the new
>>>>>>> changes at once as opposed to dealing with it twice (with package
>>>>>>>
>>>>>>> rename
>>>>>> and dependency changes) and second it will allow for a smoother
>>>>>> transition
>>>>>>
>>>>>> as the existing code will still work in a deprecated state. It will
>>>>>>>
>>>>>>> also
>>>>>> give a more consistent structure to malhar. For new operators, we
can
>>>>>> go
>>>>>> with the new package path but we need to ensure they will get moved
>>>>>> into
>>>>>> the baby packages as well.
>>>>>>
>>>>>>> I think existing operators should be renamed so that git history
>>>>>>>
>>>>>> remains. A
>>>>>
>>>>> possible solution for backward compatibility could be to subsequently
>>>>>> add
>>>>>> empty subclasses in the previous location (for existing concrete
>>>>>>
>>>>>> operators
>>>>>
>>>>> that we know are actually in use) to simplify migration for users.
>>>>>>
>>>>>>
>>>>>> Yes we can do that.
>>>>>>
>>>>>
>>>>> For demos, we can modify the paths as the apps are typically used
>>>>>
>>>>>> wholesale
>>>>>>
>>>>>> and the interface is typically manual interaction.
>>>>>>>
>>>>>>> For core, if we are adding new api subsystems, like the launcher
api
>>>>>>> we
>>>>>>> added recently for example, we can go with new package path but
if we
>>>>>>>
>>>>>>> are
>>>>>> making incremental additions to existing functionality, I feel it
is
>>>>>> better
>>>>>>
>>>>>> to keep it in the same package. I also prefer we keep the package
of
>>>>>>>
>>>>>>> the
>>>>>> implementation classes consistent with api, for understandability
and
>>>>>>
>>>>>>> readability of the code. So, for example, we don't change package
>>>>>>> path
>>>>>>>
>>>>>>> of
>>>>>> LogicalPlan as it is an implementation of DAG. It is subjective,
but
>>>>>> it
>>>>>>
>>>>>>> will be good if we can also do the same with classes closely
related
>>>>>>> to
>>>>>>>
>>>>>>> the
>>>>>>
>>>>>> implementation classes as well. Maybe we can moving these on a package
>>>>>>>
>>>>>>> by
>>>>>> package basis, like everything in com.datatorrent.stram.engine could
>>>>>> be
>>>>>>
>>>>>>> moved. For completely internal components like buffer server,
we can
>>>>>>>
>>>>>>> move
>>>>>> them wholesale. We can consider moving all api and classes, when
we go
>>>>>> to
>>>>>> next major release but would like to see if we can find a way to
>>>>>> support
>>>>>> existing api for one more major release in deprecated mode.
>>>>>>
>>>>>>>
>>>>>>> The point of the major release is to enable backward incompatible
>>>>>>>
>>>>>> changes
>>>>>> and I don't think it is realistic to support the existing API for
>>>>>> another
>>>>>> major release. IMO it is also not necessary as most existing
>>>>>> application
>>>>>> code refers to operators, attributes and the application interface.
>>>>>>
>>>>>> Perhaps
>>>>>
>>>>> it is possible to keep those around as interface extensions to help
>>>>>> migration. Custom operators may need to be migrated to reflect API
>>>>>>
>>>>>> changes,
>>>>>
>>>>> and I would consider that a reasonable task for operator developers as
>>>>>>
>>>>>> part
>>>>>
>>>>> of a major upgrade.
>>>>>>
>>>>>> It would be good if we can keep them as deprecated interface
>>>>>> extensions
>>>>>>
>>>>> for
>>>>> one release to provide a smoother transition.
>>>>>
>>>>>
>>>>> API and implementation in engine are kept separate intentionally. They
>>>>>
>>>>>> reside in different packages today, so I don't see a problem renaming
>>>>>> com.datatorrent.stram.engine as you say, even when the API cannot
be
>>>>>> touched right away.
>>>>>>
>>>>>> They are different packages but sharing a common prefix with api
will
>>>>>> be
>>>>>>
>>>>> helpful to someone new to codebase in terms of readability. Not a big
>>>>> deal
>>>>> and can be changed.
>>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise <thw@apache.org>
wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>>> This topic has come up on several PRs and I think it warrants
a
>>>>>>>>
>>>>>>>> broader
>>>>>>>
>>>>>> discussion.
>>>>>>
>>>>>>> At the time of incubation, the decision was to defer change of
Java
>>>>>>>> packages from com.datatorrent to org.apache.apex till next
major
>>>>>>>>
>>>>>>>> release
>>>>>>> to
>>>>>>>
>>>>>>> ensure backward compatibility for users.
>>>>>>>>
>>>>>>>> Unfortunately that has lead to some confusion, as contributors
>>>>>>>>
>>>>>>>> continue
>>>>>>>
>>>>>> to
>>>>>>
>>>>>>> add new code under legacy packages.
>>>>>>>>
>>>>>>>> It is also a wider issue that examples for using Apex continue
to
>>>>>>>>
>>>>>>>> refer
>>>>>>>
>>>>>> to
>>>>>>
>>>>>>> com.datatorrent packages, nearly one year after graduation. More
and
>>>>>>>>
>>>>>>>> more
>>>>>>> user code is being built on top of something that needs to change,
>>>>>>> the
>>>>>>>
>>>>>> can
>>>>>>
>>>>>>> is being kicked down the road and users will face more changes
later.
>>>>>>>>
>>>>>>>> I would like to propose the following:
>>>>>>>>
>>>>>>>> 1. All new code has to be submitted under org.apache.apex
packages
>>>>>>>>
>>>>>>>> 2. Not all code is under backward compatibility restriction
and in
>>>>>>>>
>>>>>>>> those
>>>>>>> cases we can rename the packages right away. Examples: buffer
server,
>>>>>>>
>>>>>>>> engine, demos/examples, benchmarks
>>>>>>>>
>>>>>>>> 3. Discuss when the core API and operators can be changed.
For
>>>>>>>>
>>>>>>>> operators
>>>>>>> we
>>>>>>>
>>>>>>> have a bit more freedom to do changes before a major release
as most
>>>>>>>>
>>>>>>>> of
>>>>>>>
>>>>>> them are marked @Evolving and users have the ability to continue
>>>>>>
>>>>>>> using
>>>>>>>
>>>>>> prior version of Malhar with newer engine due to engine backward
>>>>>>
>>>>>>> compatibility guarantee.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Thomas
>>>>>>>>
>>>>>>>>
>>>>>>>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message