apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Weise <...@apache.org>
Subject Re: Java packages: legacy -> org.apache.apex
Date Fri, 14 Jul 2017 16:44:02 GMT
Most of the operators evolve, as is quite clearly indicated by @Evolving
annotations. So while perhaps 0.x.x would be a more appropriate version
number, I don't think you can change that.

Thomas

On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov <v.rozov@datatorrent.com> wrote:

> If entire library is not stable, its version should be 0.x.x and there
> should not be any semantic versioning enabled or implied. It evolves. If
> some operators are stable as 3.8.x version suggests, the library should
> follow semantic versioning and it is not OK to make a major change that
> breaks semantic versioning across the entire library. It is not a finding
> an excuse not to make a change. For me semantic versioning and backward
> compatibility is more important than a proper package name.
>
> Thank you,
>
> Vlad
>
>
> On 7/14/17 09:11, Thomas Weise wrote:
>
>> Semantic versioning makes sense when you have a stable baseline. As long
>> as
>> frequent fixes need to be made to address basic issues, it makes no sense
>> to declare operators stable, which is why they are marked evolving.
>>
>> Instead of finding reasons to avoid changes that should have been made a
>> long time ago, why not discuss what a "stable operator" is and which ones
>> deserve to be in that category. Those are the ones that will get the
>> backward compatibility stub.
>>
>> Thanks,
>> Thomas
>>
>>
>> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <v.rozov@datatorrent.com>
>> wrote:
>>
>> My preference is to agree that the next Malhar release is 4.0.0 and make
>>> the proposed changes when 3.8.0 goes out. Otherwise why to keep semantic
>>> versioning checks on Malhar in case there is no version compatibility.
>>>
>>> Thank you,
>>>
>>> Vlad
>>>
>>>
>>> On 7/14/17 08:11, Thomas Weise wrote:
>>>
>>> next release is 3.8.0
>>>>
>>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <v.rozov@datatorrent.com>
>>>> wrote:
>>>>
>>>> next release - 3.9.0 or 4.0.0?
>>>>
>>>>> Thank you,
>>>>>
>>>>> Vlad
>>>>>
>>>>> On 7/13/17 22:25, Thomas Weise wrote:
>>>>>
>>>>> It is time to resurrect this thread and get going with the work.
>>>>>
>>>>>> For the next release, I will sign up to do the package move in Malhar:
>>>>>>
>>>>>> https://issues.apache.org/jira/browse/APEXMALHAR-2517
>>>>>>
>>>>>> In general this will be straightforward; most classes in Malhar are
>>>>>> marked
>>>>>> evolving and it is trivial for users to change import statements.
>>>>>> However,
>>>>>> I would suggest to discuss if there are selected operators that are
>>>>>> worth
>>>>>> keeping a backward compatibility stub in the old location.
>>>>>>
>>>>>> Here is my plan:
>>>>>>
>>>>>> 1. Relocate all classes in *lib* and *contrib* within one PR - this
is
>>>>>> all
>>>>>>
>>>>>> IDE automated work
>>>>>> 2. Add backward compatibility classes, if, any in separate PR
>>>>>> 3. Create PR for Megh library to reflect moved classes
>>>>>>
>>>>>> Thanks,
>>>>>> Thomas
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <
>>>>>> pramod@datatorrent.com
>>>>>> wrote:
>>>>>>
>>>>>> Inline
>>>>>>
>>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <thw@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>> -->
>>>>>>>
>>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
>>>>>>>> pramod@datatorrent.com
>>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> For malhar, for existing operators, I prefer we do this as
part of
>>>>>>>> the
>>>>>>>>
>>>>>>>> planned refactoring for breaking the monolith modules into
baby
>>>>>>>>>
>>>>>>>>> packages
>>>>>>>>>
>>>>>>>> and would also prefer deprecating the existing operators
in place.
>>>>>>>> Refactor into smaller modules was discussed for malhar-contrib
and
>>>>>>>> given
>>>>>>>> the overall state of that module I think it is OK to defer
package
>>>>>>>>
>>>>>>>> renaming
>>>>>>>>
>>>>>>> there. I do however prefer to see the package rename addressed
for
>>>>>>>
>>>>>>>> other
>>>>>>>> modules, especially for the main library module.
>>>>>>>>
>>>>>>>> Should we consider breaking the library into smaller modules
as
>>>>>>>> well,
>>>>>>>>
>>>>>>>> the
>>>>>>> file/block operators for example probably can be in their own
module
>>>>>>> from
>>>>>>> just an organizational perspective.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> This
>>>>>>>
>>>>>>>> will help us achieve two things. First, the user will see
all the
>>>>>>>>> new
>>>>>>>>> changes at once as opposed to dealing with it twice (with
package
>>>>>>>>>
>>>>>>>>> rename
>>>>>>>>>
>>>>>>>> and dependency changes) and second it will allow for a smoother
>>>>>>>> transition
>>>>>>>>
>>>>>>>> as the existing code will still work in a deprecated state.
It will
>>>>>>>>
>>>>>>>>> also
>>>>>>>>>
>>>>>>>> give a more consistent structure to malhar. For new operators,
we
>>>>>>>> can
>>>>>>>> go
>>>>>>>> with the new package path but we need to ensure they will
get moved
>>>>>>>> into
>>>>>>>> the baby packages as well.
>>>>>>>>
>>>>>>>> I think existing operators should be renamed so that git
history
>>>>>>>>>
>>>>>>>>> remains. A
>>>>>>>>
>>>>>>> possible solution for backward compatibility could be to subsequently
>>>>>>>
>>>>>>>> add
>>>>>>>> empty subclasses in the previous location (for existing concrete
>>>>>>>>
>>>>>>>> operators
>>>>>>>>
>>>>>>> that we know are actually in use) to simplify migration for users.
>>>>>>>
>>>>>>>>
>>>>>>>> Yes we can do that.
>>>>>>>>
>>>>>>>> For demos, we can modify the paths as the apps are typically
used
>>>>>>>
>>>>>>> wholesale
>>>>>>>>
>>>>>>>> and the interface is typically manual interaction.
>>>>>>>>
>>>>>>>>> For core, if we are adding new api subsystems, like the
launcher
>>>>>>>>> api
>>>>>>>>> we
>>>>>>>>> added recently for example, we can go with new package
path but if
>>>>>>>>> we
>>>>>>>>>
>>>>>>>>> are
>>>>>>>>>
>>>>>>>> making incremental additions to existing functionality, I
feel it is
>>>>>>>> better
>>>>>>>>
>>>>>>>> to keep it in the same package. I also prefer we keep the
package of
>>>>>>>>
>>>>>>>>> the
>>>>>>>>>
>>>>>>>> implementation classes consistent with api, for understandability
>>>>>>>> and
>>>>>>>>
>>>>>>>> readability of the code. So, for example, we don't change
package
>>>>>>>>> path
>>>>>>>>>
>>>>>>>>> of
>>>>>>>>>
>>>>>>>> LogicalPlan as it is an implementation of DAG. It is subjective,
but
>>>>>>>> it
>>>>>>>>
>>>>>>>> will be good if we can also do the same with classes closely
related
>>>>>>>>> to
>>>>>>>>>
>>>>>>>>> the
>>>>>>>>>
>>>>>>>> implementation classes as well. Maybe we can moving these
on a
>>>>>>>> package
>>>>>>>>
>>>>>>>>> by
>>>>>>>>>
>>>>>>>> package basis, like everything in com.datatorrent.stram.engine
could
>>>>>>>> be
>>>>>>>>
>>>>>>>> moved. For completely internal components like buffer server,
we can
>>>>>>>>>
>>>>>>>>> move
>>>>>>>>>
>>>>>>>> them wholesale. We can consider moving all api and classes,
when we
>>>>>>>> go
>>>>>>>> to
>>>>>>>> next major release but would like to see if we can find a
way to
>>>>>>>> support
>>>>>>>> existing api for one more major release in deprecated mode.
>>>>>>>>
>>>>>>>> The point of the major release is to enable backward incompatible
>>>>>>>>>
>>>>>>>>> changes
>>>>>>>> and I don't think it is realistic to support the existing
API for
>>>>>>>> another
>>>>>>>> major release. IMO it is also not necessary as most existing
>>>>>>>> application
>>>>>>>> code refers to operators, attributes and the application
interface.
>>>>>>>>
>>>>>>>> Perhaps
>>>>>>>>
>>>>>>> it is possible to keep those around as interface extensions to
help
>>>>>>>
>>>>>>>> migration. Custom operators may need to be migrated to reflect
API
>>>>>>>>
>>>>>>>> changes,
>>>>>>>>
>>>>>>> and I would consider that a reasonable task for operator developers
>>>>>>> as
>>>>>>>
>>>>>>>> part
>>>>>>>>
>>>>>>> of a major upgrade.
>>>>>>>
>>>>>>>> It would be good if we can keep them as deprecated interface
>>>>>>>> extensions
>>>>>>>>
>>>>>>>> for
>>>>>>> one release to provide a smoother transition.
>>>>>>>
>>>>>>>
>>>>>>> API and implementation in engine are kept separate intentionally.
>>>>>>> They
>>>>>>>
>>>>>>> reside in different packages today, so I don't see a problem
renaming
>>>>>>>> com.datatorrent.stram.engine as you say, even when the API
cannot be
>>>>>>>> touched right away.
>>>>>>>>
>>>>>>>> They are different packages but sharing a common prefix with
api
>>>>>>>> will
>>>>>>>> be
>>>>>>>>
>>>>>>>> helpful to someone new to codebase in terms of readability.
Not a
>>>>>>> big
>>>>>>> deal
>>>>>>> and can be changed.
>>>>>>>
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise <thw@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> This topic has come up on several PRs and I think it
warrants a
>>>>>>>>>>
>>>>>>>>>> broader
>>>>>>>>>>
>>>>>>>>> discussion.
>>>>>>>>
>>>>>>>> At the time of incubation, the decision was to defer change
of Java
>>>>>>>>>
>>>>>>>>>> packages from com.datatorrent to org.apache.apex
till next major
>>>>>>>>>>
>>>>>>>>>> release
>>>>>>>>>>
>>>>>>>>> to
>>>>>>>>>
>>>>>>>>> ensure backward compatibility for users.
>>>>>>>>>
>>>>>>>>>> Unfortunately that has lead to some confusion, as
contributors
>>>>>>>>>>
>>>>>>>>>> continue
>>>>>>>>>>
>>>>>>>>> to
>>>>>>>>
>>>>>>>> add new code under legacy packages.
>>>>>>>>>
>>>>>>>>>> It is also a wider issue that examples for using
Apex continue to
>>>>>>>>>>
>>>>>>>>>> refer
>>>>>>>>>>
>>>>>>>>> to
>>>>>>>>
>>>>>>>> com.datatorrent packages, nearly one year after graduation.
More and
>>>>>>>>>
>>>>>>>>>> more
>>>>>>>>>>
>>>>>>>>> user code is being built on top of something that needs
to change,
>>>>>>>>> the
>>>>>>>>>
>>>>>>>>> can
>>>>>>>>
>>>>>>>> is being kicked down the road and users will face more changes
>>>>>>>>> later.
>>>>>>>>>
>>>>>>>>>> I would like to propose the following:
>>>>>>>>>>
>>>>>>>>>> 1. All new code has to be submitted under org.apache.apex
packages
>>>>>>>>>>
>>>>>>>>>> 2. Not all code is under backward compatibility restriction
and in
>>>>>>>>>>
>>>>>>>>>> those
>>>>>>>>>>
>>>>>>>>> cases we can rename the packages right away. Examples:
buffer
>>>>>>>>> server,
>>>>>>>>>
>>>>>>>>> engine, demos/examples, benchmarks
>>>>>>>>>>
>>>>>>>>>> 3. Discuss when the core API and operators can be
changed. For
>>>>>>>>>>
>>>>>>>>>> operators
>>>>>>>>>>
>>>>>>>>> we
>>>>>>>>>
>>>>>>>>> have a bit more freedom to do changes before a major
release as
>>>>>>>>> most
>>>>>>>>>
>>>>>>>>>> of
>>>>>>>>>>
>>>>>>>>> them are marked @Evolving and users have the ability
to continue
>>>>>>>>
>>>>>>>> using
>>>>>>>>>
>>>>>>>>> prior version of Malhar with newer engine due to engine
backward
>>>>>>>>
>>>>>>>> compatibility guarantee.
>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Thomas
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message