nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <joe.w...@gmail.com>
Subject Re: NIFI-2026: Opinion needed about adding non-Apache repos
Date Thu, 16 Jun 2016 00:54:26 GMT
OK well let's keep poking at it until we have the best option possible.
Will look into sparks approach.

Thanks
Joe
On Jun 15, 2016 8:27 PM, "Andre" <andre-lists@fucs.org> wrote:

> Joe,
>
> I wouldn't say I am supportive of it... But you guys have far more
> experience in this thing than me, so I am considering all options and
> trying to keep an open mind. :-)
>
> Had it been my choice - and no side effects are caused by the changes - I
> rather use Spark's approach and have the repos @ the top level pom so that
> we can simply point users to:
>
> mvn -T2 -DskipTest -Dhadoop.version=whatever
>
> But I am happy to try to find a solution that suits the group as a whole
> and to provide the group with updates as it goes.
>
> Cheers
>
>
> On Thu, Jun 16, 2016 at 10:12 AM, Joe Witt <joe.witt@gmail.com> wrote:
>
>> Andre
>>
>> So are you also supportive of the settings based approach for now?
>>
>> Thanks
>> Joe
>> On Jun 15, 2016 8:08 PM, "Andre" <andre-lists@fucs.org> wrote:
>>
>>> Just to complement this conversation.
>>>
>>> I did use the settings.xml (basic changes, specifying profiles with the
>>> appropriate repositories)
>>>
>>> As initially suspected, the libraries are then added to NARs beyond the
>>> nifi-hadoop-libraries-bundle.
>>>
>>> Using -Pmapr -Dhadoop.version=2.7.0-mapr-1506 results in:
>>>
>>> $ find -name "*-mapr-*" | grep -v nifi-hadoop-libraries-bundle
>>>
>>> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.7.0-mapr-1506.jar
>>>
>>> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-common-2.7.0-mapr-1506.jar
>>>
>>> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-mapreduce-client-core-2.7.0-mapr-1506.jar
>>>
>>> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-common-2.7.0-mapr-1506.jar
>>>
>>> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-api-2.7.0-mapr-1506.jar
>>>
>>> ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.7.0-mapr-1506.jar
>>>
>>> ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-auth-2.7.0-mapr-1506.jar
>>>
>>> ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-common-2.7.0-mapr-1506.jar
>>>
>>> ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.7.0-mapr-1506.jar
>>>
>>> ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-mapreduce-client-core-2.7.0-mapr-1506.jar
>>>
>>> ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-common-2.7.0-mapr-1506.jar
>>>
>>> ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-api-2.7.0-mapr-1506.jar
>>>
>>>
>>> While using -Pcdh -Dhadoop.version=2.6.0-cdh5.7.1 results in:
>>>
>>> $ find -name "*-cdh*" | grep -v nifi-hadoop-libraries-bundle
>>>
>>> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-mapreduce-client-core-2.6.0-cdh5.7.1.jar
>>>
>>> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.6.0-cdh5.7.1.jar
>>>
>>> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-api-2.6.0-cdh5.7.1.jar
>>>
>>> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-common-2.6.0-cdh5.7.1.jar
>>>
>>> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-common-2.6.0-cdh5.7.1.jar
>>>
>>> ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.6.0-cdh5.7.1.jar
>>>
>>> ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-common-2.6.0-cdh5.7.1.jar
>>>
>>> ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-auth-2.6.0-cdh5.7.1.jar
>>>
>>> ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-mapreduce-client-core-2.6.0-cdh5.7.1.jar
>>>
>>> ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.6.0-cdh5.7.1.jar
>>>
>>> ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-api-2.6.0-cdh5.7.1.jar
>>>
>>> ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-common-2.6.0-cdh5.7.1.jar
>>>
>>>
>>> This doesn't seem a particularly bad thing.
>>>
>>>
>>> On Thu, Jun 16, 2016 at 1:56 AM, Andre <andre-lists@fucs.org> wrote:
>>>
>>> > All,
>>> >
>>> > Moving from GH as suggested by Matt.
>>> >
>>> >
>>> > As part of NIFI-2026 I have been experimenting with ways of using
>>> > non-vanilla Hadoop libraries when building NiFi from source.
>>> >
>>> > The idea is to streamline the overall process one must follow to get
>>> > access to HDFS compatible platforms (e.g. MapR-FS) without the need to
>>> keep
>>> > separate repositories as currently done here:
>>> > https://github.com/xmlking/mapr-nifi-hadoop-libraries-bundle
>>> >
>>> > The current commit - finally functional (thanks Matt!) - creates a
>>> profile
>>> >
>>> > custom_hadoop_libraries - more on this below.
>>> >
>>> > and two convenience properties:
>>> >
>>> > nifi.hadoop.distro.params - Used to add a java arg within
>>> > bootstrap.properties (so far only known to be required by one distro)
>>> >
>>> > nifi.hadoop.version - Used to pass vendor based version of hadoop
>>> > artifacts (e.g. 2.0.0-mr1-cdh4.2.0)
>>> >
>>> > About the profile:
>>> > - The profile enables the individual building
>>> nifi-hadoop-libraries-nar to
>>> > point mvn to CDH, HDP and MapR repos and use artifacts with the
>>> version set
>>> > by nifi.hadoop.version
>>> >
>>> > The profile is disabled by default, therefore, the profile acts as a
>>> > fencing mechanisms to ensure convenience binaries should not be tainted
>>> > with code of unknown licensing status.
>>> >
>>> > The used of profile and a custom property also ensure the use of
>>> > customised code is restricted to the points where is needed (e.g. while
>>> > HDFS libraries may be CDH, Hive libraries may still be vanilla).
>>> >
>>> >
>>> > This is not the only alternative. Joe W has suggested that we possibly
>>> > point users to alter their settings.xml (option b), while Apache Spark
>>> > simply adds the same repositories to the project main pom.xml (
>>> > https://github.com/apache/spark/blob/branch-1.6/pom.xml#L285) (option
>>> c)
>>> >
>>> >
>>> > May I ask you what is your preferred approach / comments?
>>> >
>>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message