nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sivaprasanna <sivaprasanna...@gmail.com>
Subject NIFI-5133: Guidance & help with tackling dependency version issues
Date Thu, 03 May 2018 15:51:59 GMT
Hi

I've started the initial works on implementing Google Cloud Pub/Sub
processors. The associated Jira ID is NIFI-5133
<https://issues.apache.org/jira/browse/NIFI-5133>. This will go to the
existing GCP bundle which currently has only the storage processors. Upon
some inspection, I noticed the following:

   - As of now, the bundle uses google-cloud
   <http://mvnrepository.com/artifact/com.google.cloud/google-cloud> as its
   dependency which is like uber/fat jar that contains most of the Google
   Cloud's client library SDKs including storage, bigquery, pubsub, etc. The
   main point is it is using a very older version (0.8.0)
   - I thought of using google-cloud-bom
   <http://mvnrepository.com/artifact/com.google.cloud/google-cloud-bom> in
   the bundle's POM
   <https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-gcp-bundle/pom.xml>
   and then use the required artifacts in the processor's POM
   <https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/pom.xml>.
   The benefit is, it will help us reduce the overall size of the NAR.

When I tried to do #2, I realized this is not a simple version change but a
change that brings backward compatibility issues. Ex: Some APIs used in the
older version i.e. 0.8.0 have now been entirely scrapped and moved to
different library. We can do either two things:

   1. User the Pub/Sub APIs from the older version but the problem is it's
   very old and the problem of upgrading would soon catchup with us.
   2. Or we can continue to use the older version of google-cloud-storage
   only for the storage processors and introduce the #2 mentioned above but I
   don't think then the new processors can't properly extend the existing
   AbstractGCPProcessor
   <https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/AbstractGCPProcessor.java>.


A quick glance on the processor code and the POM would help you understand
my concern.

I'm stuck up here so any help & guidance in this regard is very much
appreciated. :)

Thanks,

Sivaprasanna

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message