apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Weise <tho...@datatorrent.com>
Subject Re: Enable semantic versioning only for specific operators in Malhar
Date Wed, 23 Dec 2015 02:17:57 GMT
-1 on custom annotations.

hadoop-common is an API dependency.

[INFO] org.apache.apex:apex-api:jar:3.3.0-incubating-SNAPSHOT
[INFO] +- org.apache.hadoop:hadoop-common:jar:2.2.0:compile
[INFO] |  +- org.apache.hadoop:hadoop-annotations:jar:2.2.0:compile


On Tue, Dec 22, 2015 at 2:37 PM, Chandni Singh <chandni@datatorrent.com>
wrote:

> Pramod,
> Agreed on this point. Will create our own annotations.
>
> Thanks,
> Chandni
>
> On Tue, Dec 22, 2015 at 2:30 PM, Pramod Immaneni <pramod@datatorrent.com>
> wrote:
>
> > I suggest we don't go with hadoop annotations as that will add a direct
> > dependency from operators to hadoop even though most of them only need to
> > be dependent on apex. In future if we port apex to run on other systems
> > this will pose a problem. We either implement our own annotations or pick
> > some other independent standard.
> >
> > On Mon, Dec 14, 2015 at 10:16 PM, Chandni Singh <chandni@datatorrent.com
> >
> > wrote:
> >
> > > We need to identify the operators and components that are stable if we
> > want
> > > to go with semver check of only Stable classes.
> > > I can create an initial list.
> > >
> > > Thanks,
> > > Chandni
> > >
> > > On Mon, Dec 14, 2015 at 9:24 PM, Isha Arkatkar <isha@datatorrent.com>
> > > wrote:
> > >
> > > > Yep,  That's what I am doing now :)
> > > >
> > > > Thanks,
> > > > Isha
> > > >
> > > > On Mon, Dec 14, 2015 at 9:22 PM, Chandni Singh <
> > chandni@datatorrent.com>
> > > > wrote:
> > > >
> > > > > Isha,
> > > > >
> > > > > I think for now you can configure the japicmp plugin to exclude the
> > > > package
> > > > > as follows in the pom.xml.
> > > > >
> > > > > <excludes>
> > > > >
> > > > > <exclude>com.datatorrent.lib.parser.*</exclude>
> > > > >
> > > > > </excludes>
> > > > >
> > > > > This is an example where we can benefit from inclusion approach
> with
> > > > > japicmp 0.7 version.
> > > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Chandni
> > > > >
> > > > >
> > > > > On Mon, Dec 14, 2015 at 8:34 PM, Isha Arkatkar <
> isha@datatorrent.com
> > >
> > > > > wrote:
> > > > >
> > > > > > +1
> > > > > > When 0.7 version of japicmp is available, we can add exclusions
> for
> > > > > > @Evolving or inclusions for @Stable, whichever way is finalized.
> > > > > >
> > > > > > But before that should we add package exclusions individually
if
> > all
> > > > the
> > > > > > operators inside the package are marked Evolving?
> > > > > > I wanted to make changes to some of the parser operators in
> Malhar.
> > > But
> > > > > > changing those breaks sem version check.
> > > > > >
> > > > > > Please suggest.
> > > > > >
> > > > > > Thanks,
> > > > > > Isha
> > > > > >
> > > > > > On Sun, Dec 13, 2015 at 10:46 PM, Tushar Gosavi <
> > > > tushar@datatorrent.com>
> > > > > > wrote:
> > > > > >
> > > > > > > +1
> > > > > > > The new operators added will most likely to undergo change
> > > frequently
> > > > > > until
> > > > > > > they become stable.
> > > > > > >
> > > > > > > - Tushar.
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Dec 14, 2015 at 12:05 PM, Siyuan Hua <
> > > siyuan@datatorrent.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1
> > > > > > > >
> > > > > > > > On Sun, Dec 13, 2015 at 10:01 PM, Yogi Devendra <
> > > > > > yogidevendra@apache.org
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I am not against the Idea of using @Stable instead
of
> marking
> > > > every
> > > > > > new
> > > > > > > > > @Evolving. I agree that, it would convenient
to have
> @Stable.
> > > > > > > > >
> > > > > > > > > Just raised the point which needs further discussion,
so
> that
> > > we
> > > > > get
> > > > > > > > > suggestions from the mentors and community.
> > > > > > > > >
> > > > > > > > > ~ Yogi
> > > > > > > > >
> > > > > > > > > On 14 December 2015 at 11:23, Chandni Singh <
> > > > > chandni@datatorrent.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Can we define some criteria for deciding
when to consider
> > > > > operator
> > > > > > as
> > > > > > > > > > @Stable?
> > > > > > > > > > Yes, we should follow Hadoop's example and
formalize some
> > > > > criteria.
> > > > > > > > > >
> > > > > > > > > > [It would be difficult for an open source
project to
> track
> > > > which
> > > > > > user
> > > > > > > > is
> > > > > > > > > > using which operators. So, above strategy
may not work. ]
> > > > > > > > > > Hadoop is an open source project which actually
created
> > these
> > > > > > > > annotations
> > > > > > > > > > and it's widely used. I think any new development
takes
> > time
> > > to
> > > > > > > become
> > > > > > > > > > stable.
> > > > > > > > > > If the operators are NOT marked as @Stable,
users will
> know
> > > > that
> > > > > > when
> > > > > > > > > they
> > > > > > > > > > upgrade backward compatibility may be broken.
> > > > > > > > > > I think it has the same affect of marking
every new
> > > > > operator/class
> > > > > > as
> > > > > > > > > > @Evolving. The benefit of checking semantic
versioning of
> > > > > "Stable"
> > > > > > > > > > operators is that they are currently fewer
in number and
> > IMO
> > > > easy
> > > > > > to
> > > > > > > > > manage
> > > > > > > > > > and new development will be implicitly "Evolving".
> > > > > > > > > >
> > > > > > > > > > Chandni
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Sun, Dec 13, 2015 at 9:11 PM, Yogi Devendra
<
> > > > > > > > yogidevendra@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > When to mark certain operator as @Stable
is not clearly
> > > > > defined.
> > > > > > > > > > >
> > > > > > > > > > > Can we define some criteria for deciding
when to
> consider
> > > > > > operator
> > > > > > > as
> > > > > > > > > > > @Stable?
> > > > > > > > > > >
> > > > > > > > > > > For example one criteria could be,
if operator is
> running
> > > for
> > > > > >1
> > > > > > > year
> > > > > > > > > in
> > > > > > > > > > > production environment for some user.
Can we come with
> > some
> > > > > > > strategy
> > > > > > > > > like
> > > > > > > > > > > this?
> > > > > > > > > > > [It would be difficult for an open
source project to
> > track
> > > > > which
> > > > > > > user
> > > > > > > > > is
> > > > > > > > > > > using which operators. So, above strategy
may not
> work. ]
> > > > > > > > > > >
> > > > > > > > > > > ~ Yogi
> > > > > > > > > > >
> > > > > > > > > > > On 14 December 2015 at 05:42, Timothy
Farkas <
> > > > > > tim@datatorrent.com>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > +1
> > > > > > > > > > > > On Dec 13, 2015 4:08 PM, "Chandni
Singh" <
> > > > > > > chandni@datatorrent.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi,
> > > > > > > > > > > > >
> > > > > > > > > > > > > In Malhar there are  relatively
smaller number of
> > > > > operators
> > > > > > > that
> > > > > > > > > we
> > > > > > > > > > > use
> > > > > > > > > > > > in
> > > > > > > > > > > > > our demo applications, customer
applications, POCs
> > etc
> > > > that
> > > > > > are
> > > > > > > > > > mature.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The library is cluttered
with operators especially
> in
> > > > > > lib/util,
> > > > > > > > > > > lib/algo,
> > > > > > > > > > > > > lib/math packages which can
be cleaned up by either
> > > > > removing
> > > > > > > them
> > > > > > > > > or
> > > > > > > > > > > > > improving them but that breaks
semantic versioning.
> > > > > > > > > > > > >
> > > > > > > > > > > > > When we add new operators/utilities
it takes
> certain
> > > time
> > > > > for
> > > > > > > > them
> > > > > > > > > to
> > > > > > > > > > > > > mature. Japicmp doesn't help
because it doesn't
> honor
> > > > > > @Evolving
> > > > > > > > > > > @Unstable
> > > > > > > > > > > > > annotations for now.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I wanted to propose that
we add an annotation,
> let's
> > > say,
> > > > > > > re-use
> > > > > > > > > > > hadoop's
> > > > > > > > > > > > > @Stable and mark the operators
which are stable
> with
> > it
> > > > and
> > > > > > > > perform
> > > > > > > > > > > > semver
> > > > > > > > > > > > > check on just these operators.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The 0.7.0 version of japi
cmp has the support for
> > > > > inclusions
> > > > > > > (as
> > > > > > > > > well
> > > > > > > > > > > as
> > > > > > > > > > > > > exclusions) based on annotations.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Here is the info:
> > > > > > > > > > > > > https://github.com/siom79/japicmp/issues/88
> > > > > > > > > > > > >
> > > > > > > > > > > > > The reason I am inclined
to the inclusion approach
> is
> > > > that
> > > > > > > there
> > > > > > > > > are
> > > > > > > > > > > > > relatively smaller number
of operators which IMO
> are
> > > > > stable.
> > > > > > A
> > > > > > > > lot
> > > > > > > > > of
> > > > > > > > > > > > them
> > > > > > > > > > > > > aren't.
> > > > > > > > > > > > > So instead of going and marking
so many as
> Evolving,
> > we
> > > > > will
> > > > > > > mark
> > > > > > > > > > > > > relatively few of them as
stable.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Also new development can
be facilitated by this. We
> > > > > wouldn't
> > > > > > > have
> > > > > > > > > to
> > > > > > > > > > > add
> > > > > > > > > > > > > @Evolving to everything which
is new. Instead we
> will
> > > > mark
> > > > > it
> > > > > > > > > @Stable
> > > > > > > > > > > > when
> > > > > > > > > > > > > it is.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Please let me know what you
think?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Chandni
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message