apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Weise <...@apache.org>
Subject Re: Java packages: legacy -> org.apache.apex
Date Fri, 14 Jul 2017 15:11:33 GMT
next release is 3.8.0

On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <v.rozov@datatorrent.com> wrote:

> next release - 3.9.0 or 4.0.0?
>
> Thank you,
>
> Vlad
>
> On 7/13/17 22:25, Thomas Weise wrote:
>
>> It is time to resurrect this thread and get going with the work.
>>
>> For the next release, I will sign up to do the package move in Malhar:
>>
>> https://issues.apache.org/jira/browse/APEXMALHAR-2517
>>
>> In general this will be straightforward; most classes in Malhar are marked
>> evolving and it is trivial for users to change import statements. However,
>> I would suggest to discuss if there are selected operators that are worth
>> keeping a backward compatibility stub in the old location.
>>
>> Here is my plan:
>>
>> 1. Relocate all classes in *lib* and *contrib* within one PR - this is all
>>
>> IDE automated work
>> 2. Add backward compatibility classes, if, any in separate PR
>> 3. Create PR for Megh library to reflect moved classes
>>
>> Thanks,
>> Thomas
>>
>>
>>
>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <pramod@datatorrent.com>
>> wrote:
>>
>> Inline
>>>
>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <thw@apache.org> wrote:
>>>
>>> -->
>>>>
>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
>>>> pramod@datatorrent.com
>>>>
>>>> wrote:
>>>>
>>>> For malhar, for existing operators, I prefer we do this as part of the
>>>>> planned refactoring for breaking the monolith modules into baby
>>>>>
>>>> packages
>>>
>>>> and would also prefer deprecating the existing operators in place.
>>>>>
>>>>
>>>> Refactor into smaller modules was discussed for malhar-contrib and given
>>>> the overall state of that module I think it is OK to defer package
>>>>
>>> renaming
>>>
>>>> there. I do however prefer to see the package rename addressed for other
>>>> modules, especially for the main library module.
>>>>
>>>> Should we consider breaking the library into smaller modules as well,
>>> the
>>> file/block operators for example probably can be in their own module from
>>> just an organizational perspective.
>>>
>>>
>>>
>>>> This
>>>>> will help us achieve two things. First, the user will see all the new
>>>>> changes at once as opposed to dealing with it twice (with package
>>>>>
>>>> rename
>>>
>>>> and dependency changes) and second it will allow for a smoother
>>>>>
>>>> transition
>>>>
>>>>> as the existing code will still work in a deprecated state. It will
>>>>>
>>>> also
>>>
>>>> give a more consistent structure to malhar. For new operators, we can
>>>>>
>>>> go
>>>
>>>> with the new package path but we need to ensure they will get moved
>>>>>
>>>> into
>>>
>>>> the baby packages as well.
>>>>>
>>>>> I think existing operators should be renamed so that git history
>>>>
>>> remains. A
>>>
>>>> possible solution for backward compatibility could be to subsequently
>>>> add
>>>> empty subclasses in the previous location (for existing concrete
>>>>
>>> operators
>>>
>>>> that we know are actually in use) to simplify migration for users.
>>>>
>>>>
>>>> Yes we can do that.
>>>
>>>
>>> For demos, we can modify the paths as the apps are typically used
>>>>>
>>>> wholesale
>>>>
>>>>> and the interface is typically manual interaction.
>>>>>
>>>>> For core, if we are adding new api subsystems, like the launcher api
we
>>>>> added recently for example, we can go with new package path but if we
>>>>>
>>>> are
>>>
>>>> making incremental additions to existing functionality, I feel it is
>>>>>
>>>> better
>>>>
>>>>> to keep it in the same package. I also prefer we keep the package of
>>>>>
>>>> the
>>>
>>>> implementation classes consistent with api, for understandability and
>>>>> readability of the code. So, for example, we don't change package path
>>>>>
>>>> of
>>>
>>>> LogicalPlan as it is an implementation of DAG. It is subjective, but it
>>>>> will be good if we can also do the same with classes closely related
to
>>>>>
>>>> the
>>>>
>>>>> implementation classes as well. Maybe we can moving these on a package
>>>>>
>>>> by
>>>
>>>> package basis, like everything in com.datatorrent.stram.engine could be
>>>>> moved. For completely internal components like buffer server, we can
>>>>>
>>>> move
>>>
>>>> them wholesale. We can consider moving all api and classes, when we go
>>>>>
>>>> to
>>>
>>>> next major release but would like to see if we can find a way to
>>>>>
>>>> support
>>>
>>>> existing api for one more major release in deprecated mode.
>>>>>
>>>>>
>>>>> The point of the major release is to enable backward incompatible
>>>> changes
>>>> and I don't think it is realistic to support the existing API for
>>>> another
>>>> major release. IMO it is also not necessary as most existing application
>>>> code refers to operators, attributes and the application interface.
>>>>
>>> Perhaps
>>>
>>>> it is possible to keep those around as interface extensions to help
>>>> migration. Custom operators may need to be migrated to reflect API
>>>>
>>> changes,
>>>
>>>> and I would consider that a reasonable task for operator developers as
>>>>
>>> part
>>>
>>>> of a major upgrade.
>>>>
>>>> It would be good if we can keep them as deprecated interface extensions
>>> for
>>> one release to provide a smoother transition.
>>>
>>>
>>> API and implementation in engine are kept separate intentionally. They
>>>> reside in different packages today, so I don't see a problem renaming
>>>> com.datatorrent.stram.engine as you say, even when the API cannot be
>>>> touched right away.
>>>>
>>>> They are different packages but sharing a common prefix with api will be
>>> helpful to someone new to codebase in terms of readability. Not a big
>>> deal
>>> and can be changed.
>>>
>>>
>>> Thanks
>>>>
>>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise <thw@apache.org>
wrote:
>>>>>
>>>>> Hi,
>>>>>>
>>>>>> This topic has come up on several PRs and I think it warrants a
>>>>>>
>>>>> broader
>>>
>>>> discussion.
>>>>>>
>>>>>> At the time of incubation, the decision was to defer change of Java
>>>>>> packages from com.datatorrent to org.apache.apex till next major
>>>>>>
>>>>> release
>>>>
>>>>> to
>>>>>
>>>>>> ensure backward compatibility for users.
>>>>>>
>>>>>> Unfortunately that has lead to some confusion, as contributors
>>>>>>
>>>>> continue
>>>
>>>> to
>>>>>
>>>>>> add new code under legacy packages.
>>>>>>
>>>>>> It is also a wider issue that examples for using Apex continue to
>>>>>>
>>>>> refer
>>>
>>>> to
>>>>>
>>>>>> com.datatorrent packages, nearly one year after graduation. More
and
>>>>>>
>>>>> more
>>>>
>>>>> user code is being built on top of something that needs to change,
>>>>>>
>>>>> the
>>>
>>>> can
>>>>>
>>>>>> is being kicked down the road and users will face more changes later.
>>>>>>
>>>>>> I would like to propose the following:
>>>>>>
>>>>>> 1. All new code has to be submitted under org.apache.apex packages
>>>>>>
>>>>>> 2. Not all code is under backward compatibility restriction and in
>>>>>>
>>>>> those
>>>>
>>>>> cases we can rename the packages right away. Examples: buffer server,
>>>>>> engine, demos/examples, benchmarks
>>>>>>
>>>>>> 3. Discuss when the core API and operators can be changed. For
>>>>>>
>>>>> operators
>>>>
>>>>> we
>>>>>
>>>>>> have a bit more freedom to do changes before a major release as most
>>>>>>
>>>>> of
>>>
>>>> them are marked @Evolving and users have the ability to continue
>>>>>>
>>>>> using
>>>
>>>> prior version of Malhar with newer engine due to engine backward
>>>>>> compatibility guarantee.
>>>>>>
>>>>>> Thanks,
>>>>>> Thomas
>>>>>>
>>>>>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message