apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vlad Rozov <v.ro...@datatorrent.com>
Subject Re: Java packages: legacy -> org.apache.apex
Date Fri, 14 Jul 2017 14:55:15 GMT
next release - 3.9.0 or 4.0.0?

Thank you,


On 7/13/17 22:25, Thomas Weise wrote:
> It is time to resurrect this thread and get going with the work.
> For the next release, I will sign up to do the package move in Malhar:
> https://issues.apache.org/jira/browse/APEXMALHAR-2517
> In general this will be straightforward; most classes in Malhar are marked
> evolving and it is trivial for users to change import statements. However,
> I would suggest to discuss if there are selected operators that are worth
> keeping a backward compatibility stub in the old location.
> Here is my plan:
> 1. Relocate all classes in *lib* and *contrib* within one PR - this is all
> IDE automated work
> 2. Add backward compatibility classes, if, any in separate PR
> 3. Create PR for Megh library to reflect moved classes
> Thanks,
> Thomas
> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <pramod@datatorrent.com>
> wrote:
>> Inline
>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <thw@apache.org> wrote:
>>> -->
>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <pramod@datatorrent.com
>>> wrote:
>>>> For malhar, for existing operators, I prefer we do this as part of the
>>>> planned refactoring for breaking the monolith modules into baby
>> packages
>>>> and would also prefer deprecating the existing operators in place.
>>> Refactor into smaller modules was discussed for malhar-contrib and given
>>> the overall state of that module I think it is OK to defer package
>> renaming
>>> there. I do however prefer to see the package rename addressed for other
>>> modules, especially for the main library module.
>> Should we consider breaking the library into smaller modules as well, the
>> file/block operators for example probably can be in their own module from
>> just an organizational perspective.
>>>> This
>>>> will help us achieve two things. First, the user will see all the new
>>>> changes at once as opposed to dealing with it twice (with package
>> rename
>>>> and dependency changes) and second it will allow for a smoother
>>> transition
>>>> as the existing code will still work in a deprecated state. It will
>> also
>>>> give a more consistent structure to malhar. For new operators, we can
>> go
>>>> with the new package path but we need to ensure they will get moved
>> into
>>>> the baby packages as well.
>>> I think existing operators should be renamed so that git history
>> remains. A
>>> possible solution for backward compatibility could be to subsequently add
>>> empty subclasses in the previous location (for existing concrete
>> operators
>>> that we know are actually in use) to simplify migration for users.
>> Yes we can do that.
>>>> For demos, we can modify the paths as the apps are typically used
>>> wholesale
>>>> and the interface is typically manual interaction.
>>>> For core, if we are adding new api subsystems, like the launcher api we
>>>> added recently for example, we can go with new package path but if we
>> are
>>>> making incremental additions to existing functionality, I feel it is
>>> better
>>>> to keep it in the same package. I also prefer we keep the package of
>> the
>>>> implementation classes consistent with api, for understandability and
>>>> readability of the code. So, for example, we don't change package path
>> of
>>>> LogicalPlan as it is an implementation of DAG. It is subjective, but it
>>>> will be good if we can also do the same with classes closely related to
>>> the
>>>> implementation classes as well. Maybe we can moving these on a package
>> by
>>>> package basis, like everything in com.datatorrent.stram.engine could be
>>>> moved. For completely internal components like buffer server, we can
>> move
>>>> them wholesale. We can consider moving all api and classes, when we go
>> to
>>>> next major release but would like to see if we can find a way to
>> support
>>>> existing api for one more major release in deprecated mode.
>>> The point of the major release is to enable backward incompatible changes
>>> and I don't think it is realistic to support the existing API for another
>>> major release. IMO it is also not necessary as most existing application
>>> code refers to operators, attributes and the application interface.
>> Perhaps
>>> it is possible to keep those around as interface extensions to help
>>> migration. Custom operators may need to be migrated to reflect API
>> changes,
>>> and I would consider that a reasonable task for operator developers as
>> part
>>> of a major upgrade.
>> It would be good if we can keep them as deprecated interface extensions for
>> one release to provide a smoother transition.
>>> API and implementation in engine are kept separate intentionally. They
>>> reside in different packages today, so I don't see a problem renaming
>>> com.datatorrent.stram.engine as you say, even when the API cannot be
>>> touched right away.
>> They are different packages but sharing a common prefix with api will be
>> helpful to someone new to codebase in terms of readability. Not a big deal
>> and can be changed.
>>> Thanks
>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise <thw@apache.org> wrote:
>>>>> Hi,
>>>>> This topic has come up on several PRs and I think it warrants a
>> broader
>>>>> discussion.
>>>>> At the time of incubation, the decision was to defer change of Java
>>>>> packages from com.datatorrent to org.apache.apex till next major
>>> release
>>>> to
>>>>> ensure backward compatibility for users.
>>>>> Unfortunately that has lead to some confusion, as contributors
>> continue
>>>> to
>>>>> add new code under legacy packages.
>>>>> It is also a wider issue that examples for using Apex continue to
>> refer
>>>> to
>>>>> com.datatorrent packages, nearly one year after graduation. More and
>>> more
>>>>> user code is being built on top of something that needs to change,
>> the
>>>> can
>>>>> is being kicked down the road and users will face more changes later.
>>>>> I would like to propose the following:
>>>>> 1. All new code has to be submitted under org.apache.apex packages
>>>>> 2. Not all code is under backward compatibility restriction and in
>>> those
>>>>> cases we can rename the packages right away. Examples: buffer server,
>>>>> engine, demos/examples, benchmarks
>>>>> 3. Discuss when the core API and operators can be changed. For
>>> operators
>>>> we
>>>>> have a bit more freedom to do changes before a major release as most
>> of
>>>>> them are marked @Evolving and users have the ability to continue
>> using
>>>>> prior version of Malhar with newer engine due to engine backward
>>>>> compatibility guarantee.
>>>>> Thanks,
>>>>> Thomas

View raw message