nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <joe.w...@gmail.com>
Subject Re: NIFI-2026: Opinion needed about adding non-Apache repos
Date Thu, 16 Jun 2016 00:12:54 GMT
Andre

So are you also supportive of the settings based approach for now?

Thanks
Joe
On Jun 15, 2016 8:08 PM, "Andre" <andre-lists@fucs.org> wrote:

> Just to complement this conversation.
>
> I did use the settings.xml (basic changes, specifying profiles with the
> appropriate repositories)
>
> As initially suspected, the libraries are then added to NARs beyond the
> nifi-hadoop-libraries-bundle.
>
> Using -Pmapr -Dhadoop.version=2.7.0-mapr-1506 results in:
>
> $ find -name "*-mapr-*" | grep -v nifi-hadoop-libraries-bundle
>
> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.7.0-mapr-1506.jar
>
> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-common-2.7.0-mapr-1506.jar
>
> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-mapreduce-client-core-2.7.0-mapr-1506.jar
>
> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-common-2.7.0-mapr-1506.jar
>
> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-api-2.7.0-mapr-1506.jar
>
> ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.7.0-mapr-1506.jar
>
> ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-auth-2.7.0-mapr-1506.jar
>
> ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-common-2.7.0-mapr-1506.jar
>
> ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.7.0-mapr-1506.jar
>
> ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-mapreduce-client-core-2.7.0-mapr-1506.jar
>
> ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-common-2.7.0-mapr-1506.jar
>
> ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-api-2.7.0-mapr-1506.jar
>
>
> While using -Pcdh -Dhadoop.version=2.6.0-cdh5.7.1 results in:
>
> $ find -name "*-cdh*" | grep -v nifi-hadoop-libraries-bundle
>
> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-mapreduce-client-core-2.6.0-cdh5.7.1.jar
>
> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.6.0-cdh5.7.1.jar
>
> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-api-2.6.0-cdh5.7.1.jar
>
> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-common-2.6.0-cdh5.7.1.jar
>
> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-common-2.6.0-cdh5.7.1.jar
>
> ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.6.0-cdh5.7.1.jar
>
> ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-common-2.6.0-cdh5.7.1.jar
>
> ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-auth-2.6.0-cdh5.7.1.jar
>
> ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-mapreduce-client-core-2.6.0-cdh5.7.1.jar
>
> ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.6.0-cdh5.7.1.jar
>
> ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-api-2.6.0-cdh5.7.1.jar
>
> ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-common-2.6.0-cdh5.7.1.jar
>
>
> This doesn't seem a particularly bad thing.
>
>
> On Thu, Jun 16, 2016 at 1:56 AM, Andre <andre-lists@fucs.org> wrote:
>
> > All,
> >
> > Moving from GH as suggested by Matt.
> >
> >
> > As part of NIFI-2026 I have been experimenting with ways of using
> > non-vanilla Hadoop libraries when building NiFi from source.
> >
> > The idea is to streamline the overall process one must follow to get
> > access to HDFS compatible platforms (e.g. MapR-FS) without the need to
> keep
> > separate repositories as currently done here:
> > https://github.com/xmlking/mapr-nifi-hadoop-libraries-bundle
> >
> > The current commit - finally functional (thanks Matt!) - creates a
> profile
> >
> > custom_hadoop_libraries - more on this below.
> >
> > and two convenience properties:
> >
> > nifi.hadoop.distro.params - Used to add a java arg within
> > bootstrap.properties (so far only known to be required by one distro)
> >
> > nifi.hadoop.version - Used to pass vendor based version of hadoop
> > artifacts (e.g. 2.0.0-mr1-cdh4.2.0)
> >
> > About the profile:
> > - The profile enables the individual building nifi-hadoop-libraries-nar
> to
> > point mvn to CDH, HDP and MapR repos and use artifacts with the version
> set
> > by nifi.hadoop.version
> >
> > The profile is disabled by default, therefore, the profile acts as a
> > fencing mechanisms to ensure convenience binaries should not be tainted
> > with code of unknown licensing status.
> >
> > The used of profile and a custom property also ensure the use of
> > customised code is restricted to the points where is needed (e.g. while
> > HDFS libraries may be CDH, Hive libraries may still be vanilla).
> >
> >
> > This is not the only alternative. Joe W has suggested that we possibly
> > point users to alter their settings.xml (option b), while Apache Spark
> > simply adds the same repositories to the project main pom.xml (
> > https://github.com/apache/spark/blob/branch-1.6/pom.xml#L285) (option c)
> >
> >
> > May I ask you what is your preferred approach / comments?
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message