apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chetan Narsude <che...@datatorrent.com>
Subject Re: More sensible modules/artifacts in malhar
Date Tue, 22 Sep 2015 18:20:26 GMT
+1

On Tue, Sep 22, 2015 at 11:08 AM, Gaurav Gupta <gaurav@datatorrent.com>
wrote:

> I agree with David.. Each artifact should have it's own version
>
> Thanks
> -Gaurav
>
> On Tue, Sep 22, 2015 at 11:07 AM, David Yan <david@datatorrent.com> wrote:
>
> > I actually think that each baby artifact should have its own version,
> > because each artifact has its own interface and its own life cycle,
> > especially after we break up the giant library, applications will depend
> on
> > the baby artifacts instead of the giant library.  For example if there is
> > no change in malhar-contrib-kafka (I think the name should actually be
> > apex-malhar-kafka), we should not confuse users by bumping the version.
> >
> > David
> >
> > On Tue, Sep 22, 2015 at 9:03 AM, Andy Perlitch <andy@datatorrent.com>
> > wrote:
> >
> > > Tushar,
> > >
> > > I agree that all modules should inherit the version from the "parent
> pom"
> > > of the malhar repo. I think the benefits outweigh the cost of bumping
> > > versions of components that haven't actually changed. I'd love to get
> > > others feedback on this as well.
> > >
> > > On another note, I plan on starting a spreadsheet/googledoc with the
> > > possible groupings of operators into these modules. Stay tuned...
> > >
> > > -Andy
> > >
> > > On Mon, Sep 21, 2015 at 11:51 PM, Tushar Gosavi <
> tushar@datatorrent.com>
> > > wrote:
> > >
> > > > +1 for the general idea
> > > >
> > > > Does these independent modules going to have independent versions?
> For
> > > > example, if there is no change in kafka operator between malhar 3.0
> and
> > > > malhar 4.0, will we increment version of malhar-contrib-kafka to
> 4.0. I
> > > > have learned from my previous project that, It is easier to manage
> > > versions
> > > > if we make all modules at same version level for a release, even if
> > there
> > > > is no change in a particular module.
> > > >
> > > > - Tushar.
> > > >
> > > >
> > > >
> > > > On Fri, Sep 18, 2015 at 12:18 AM, Timothy Farkas <
> tim@datatorrent.com>
> > > > wrote:
> > > >
> > > > > I agree Andy's solution is better, but just for the sake of
> argument
> > > > > profiles can be inherited from a parent pom, so if the maven
> > archetype
> > > > > defines a new project with a parent pom with the correct profiles
> > > > defined,
> > > > > then the desired profiles can be activated in the pom of the new
> > > project.
> > > > > It is no more complicated than adding additional dependencies to
> your
> > > > > project.
> > > > >
> > > > > On Thu, Sep 17, 2015 at 10:32 AM, Sandesh Hegde <
> > > sandesh@datatorrent.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Currently all the dependencies in Malhar-Contrib are marked
as
> > > > optional.
> > > > > So
> > > > > > users have to already modify the existing POM to use it in their
> > > > project.
> > > > > > So restructuring should be fine.
> > > > > >
> > > > > > On Thu, Sep 17, 2015 at 11:29 AM Chetan Narsude <
> > > > chetan@datatorrent.com>
> > > > > > wrote:
> > > > > >
> > > > > > > The profiles are excellent when you are developing
> > malhar-contrib.
> > > > > > Profiles
> > > > > > > do not work when you are using malhar-contrib. The problem
Andy
> > is
> > > > > > trying
> > > > > > > to solve is the later. If there is an elegant solution
which I
> am
> > > > > missing
> > > > > > > using profiles, please correct me.
> > > > > > >
> > > > > > > The way Andy suggested is the way many successful projects
do
> it.
> > > > Look
> > > > > at
> > > > > > > Netty as an example.
> > > > > > >
> > > > > > > +1 for that.
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Chetan
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Sep 17, 2015 at 11:22 AM, Timothy Farkas <
> > > > tim@datatorrent.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I think restructuring the project in that way would
be the
> > > > > technically
> > > > > > > > correct thing to do, but if people are unwilling to
accept
> the
> > > > change
> > > > > > in
> > > > > > > > project structure you could achieve something similar
by
> using
> > > > maven
> > > > > > > > profiles. With profiles the project structure would
remain as
> > is.
> > > > > > > Profiles
> > > > > > > > could be added to the malhar pom, and a profile would
define
> > the
> > > > > > > > dependencies needed for different types of operators.
For
> > example
> > > > the
> > > > > > > hbase
> > > > > > > > profile would define the dependencies for the hbase
operator.
> > > Then
> > > > > any
> > > > > > > > project using a malhar library would just activate
the
> correct
> > > > > profile
> > > > > > in
> > > > > > > > it's pom, and the correct dependencies would be pulled
in.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://maven.apache.org/guides/introduction/introduction-to-profiles.html
> > > > > > > >
> > > > > > > > On Thu, Sep 17, 2015 at 10:01 AM, Andy Perlitch <
> > > > > andy@datatorrent.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi everyone,
> > > > > > > > >
> > > > > > > > > I am currently assigned to MLHR-1843
> > > > > > > > > <https://malhar.atlassian.net/browse/MLHR-1843>,
which
> > > > essentially
> > > > > > > aims
> > > > > > > > to
> > > > > > > > > expose smaller, more consumable maven artifacts
that would
> do
> > > > away
> > > > > > with
> > > > > > > > the
> > > > > > > > > need to manually include necessary dependencies
based on
> the
> > > > > > operators
> > > > > > > in
> > > > > > > > > use.
> > > > > > > > >
> > > > > > > > > As an example, say I am building an app package
that needs
> > > Kafka
> > > > > > input
> > > > > > > > and
> > > > > > > > > output operators, but I don't want all the other
transitive
> > > > > > > dependencies
> > > > > > > > > that come via malhar-contrib. Currently I would
need to
> > specify
> > > > > > > > > malhar-contrib as a dependency, and add an exclusions
block
> > in
> > > > my
> > > > > > app
> > > > > > > > > package pom:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *<dependency>  <groupId>com.datatorrent</groupId>
> > > > > > > > > <artifactId>malhar-contrib</artifactId>
> > > <version>3.0.0</version>
> > > > > > <!--
> > > > > > > > so
> > > > > > > > > none of malhar-contrib's deps are included -->*
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *  <exclusions>    <exclusion>  
   <groupId>*</groupId>
> > > > > > > > > <artifactId>*</artifactId>    </exclusion>
> > > > > > </exclusions></dependency>*
> > > > > > > > >
> > > > > > > > > Then, I would have to include the kafka library
explicitly
> > as a
> > > > > > > > dependency:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *<dependency>  <groupId>org.apache.kafka</groupId>
> > > > > > > > > <artifactId>kafka_2.10</artifactId>
> > > > > > > > > <version>0.8.1.1</version></dependency>*
> > > > > > > > >
> > > > > > > > > Wouldn't it be nice if I could just put this
in my pom?:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *<dependency>  <groupId>com.datatorrent</groupId>
> > > > > > > > > <artifactId>malhar-contrib-kafka</artifactId>
> > > > > > > > > <version>3.0.0</version></dependency>*
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > In order to make this possible, we will need
to organize
> the
> > > > malhar
> > > > > > > > project
> > > > > > > > > into more granular modules (artifacts). Specifically,
the
> > > > > > > malhar-contrib
> > > > > > > > > artifact would essentially just be a pom that
specifies
> each
> > > > > smaller
> > > > > > > > module
> > > > > > > > > as a dependency:
> > > > > > > > >
> > > > > > > > > *<!-- in malhar-contrib's pom.xml: -->*
> > > > > > > > >
> > > > > > > > > *<modules>  <module>kafka</module>*
> > > > > > > > > *  <module>twitter</module>*
> > > > > > > > > *  <module>redis</module>*
> > > > > > > > >
> > > > > > > > > *  <!-- other smaller modules --></modules>*
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *<dependency>  <groupId>com.datatorrent</groupId>
> > > > > > > > > <artifactId>malhar-contrib-kafka</artifactId>
> > > > > > > > > <version>3.0.0</version></dependency>*
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *<dependency>  <groupId>com.datatorrent</groupId>
> > > > > > > > > <artifactId>malhar-contrib-twitter</artifactId>
> > > > > > > > > <version>3.0.0</version></dependency>*
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *<dependency>  <groupId>com.datatorrent</groupId>
> > > > > > > > > <artifactId>malhar-contrib-redis</artifactId>
> > > > > > > > > <version>3.0.0</version></dependency>*
> > > > > > > > >
> > > > > > > > > With these changes, there may be a risk of breaking
> backwards
> > > > > > > > > compatibility, however I think the gain in usability
of
> > malhar
> > > > > merits
> > > > > > > the
> > > > > > > > > effort to make this work.
> > > > > > > > >
> > > > > > > > > I am still relatively new to maven, so I would
love to get
> > some
> > > > > > > feedback
> > > > > > > > > from other devs about this!
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Regards,
> > > > > > > > > Andy Perlitch
> > > > > > > > > Software Engineer
> > > > > > > > > DataTorrent Inc
> > > > > > > > > (408)829-9319
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Andy Perlitch
> > > Software Engineer
> > > DataTorrent Inc
> > > (408)829-9319
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message