apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timothy Farkas <...@datatorrent.com>
Subject Re: More sensible modules/artifacts in malhar
Date Thu, 17 Sep 2015 18:22:05 GMT
I think restructuring the project in that way would be the technically
correct thing to do, but if people are unwilling to accept the change in
project structure you could achieve something similar by using maven
profiles. With profiles the project structure would remain as is. Profiles
could be added to the malhar pom, and a profile would define the
dependencies needed for different types of operators. For example the hbase
profile would define the dependencies for the hbase operator. Then any
project using a malhar library would just activate the correct profile in
it's pom, and the correct dependencies would be pulled in.

http://maven.apache.org/guides/introduction/introduction-to-profiles.html

On Thu, Sep 17, 2015 at 10:01 AM, Andy Perlitch <andy@datatorrent.com>
wrote:

> Hi everyone,
>
> I am currently assigned to MLHR-1843
> <https://malhar.atlassian.net/browse/MLHR-1843>, which essentially aims to
> expose smaller, more consumable maven artifacts that would do away with the
> need to manually include necessary dependencies based on the operators in
> use.
>
> As an example, say I am building an app package that needs Kafka input and
> output operators, but I don't want all the other transitive dependencies
> that come via malhar-contrib. Currently I would need to specify
> malhar-contrib as a dependency, and add an exclusions block  in my app
> package pom:
>
>
>
>
>
> *<dependency>  <groupId>com.datatorrent</groupId>
> <artifactId>malhar-contrib</artifactId>  <version>3.0.0</version>
 <!-- so
> none of malhar-contrib's deps are included -->*
>
>
>
>
>
>
> *  <exclusions>    <exclusion>      <groupId>*</groupId>
> <artifactId>*</artifactId>    </exclusion>  </exclusions></dependency>*
>
> Then, I would have to include the kafka library explicitly as a dependency:
>
>
>
>
>
> *<dependency>  <groupId>org.apache.kafka</groupId>
> <artifactId>kafka_2.10</artifactId>
> <version>0.8.1.1</version></dependency>*
>
> Wouldn't it be nice if I could just put this in my pom?:
>
>
>
>
>
> *<dependency>  <groupId>com.datatorrent</groupId>
> <artifactId>malhar-contrib-kafka</artifactId>
> <version>3.0.0</version></dependency>*
>
>
> In order to make this possible, we will need to organize the malhar project
> into more granular modules (artifacts). Specifically, the malhar-contrib
> artifact would essentially just be a pom that specifies each smaller module
> as a dependency:
>
> *<!-- in malhar-contrib's pom.xml: -->*
>
> *<modules>  <module>kafka</module>*
> *  <module>twitter</module>*
> *  <module>redis</module>*
>
> *  <!-- other smaller modules --></modules>*
>
>
>
>
> *<dependency>  <groupId>com.datatorrent</groupId>
> <artifactId>malhar-contrib-kafka</artifactId>
> <version>3.0.0</version></dependency>*
>
>
>
>
> *<dependency>  <groupId>com.datatorrent</groupId>
> <artifactId>malhar-contrib-twitter</artifactId>
> <version>3.0.0</version></dependency>*
>
>
>
>
> *<dependency>  <groupId>com.datatorrent</groupId>
> <artifactId>malhar-contrib-redis</artifactId>
> <version>3.0.0</version></dependency>*
>
> With these changes, there may be a risk of breaking backwards
> compatibility, however I think the gain in usability of malhar merits the
> effort to make this work.
>
> I am still relatively new to maven, so I would love to get some feedback
> from other devs about this!
>
> --
> Regards,
> Andy Perlitch
> Software Engineer
> DataTorrent Inc
> (408)829-9319
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message