nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Grande <>
Subject Re: [DISCUSS] Scale-out/Object Storage - taming the diversity of processors
Date Tue, 21 Feb 2017 23:21:03 GMT

I came across multiple NiFi use cases where going through the HDFS layer
and the fs plugin may not be possible. I.e. when no HDFS layer present at
all, so no NN to connect to.

Another important aspect is operations. Current PutHDFS model with
additional jar location, well, it kinda works, but I very much dislike it.
Too many possibilities for a human error in addition to deployment pain,
especially in a cluster.

Finally, native object storage processors have features which may not even
apply to the HDFS layer. E.g. the Azure storage has Table storage, etc.

I agree consolidating various efforts is worthwhile, but only within a
context of a specific storage solution. Not 'unifying' them into a single


On Tue, Feb 21, 2017, 6:10 PM Andre <> wrote:

> dev,
> I was having a chat with Pierre around PR#379 and we thought it would be
> worth sharing this with the wider group:
> I recently noticed that we merged a number of PRs and merges around
> scale-out/cloud based object store into the master.
> Would it make sense to start considering adopting a pattern where
> Put/Get/ListHDFS are used in tandem with implementations of the
> hadoop.filesystem interfaces instead of creating new processors, except
> where a particular deficiency/incompatibility in the hadoop.filesystem
> implementation exists?
> Candidates for removal / non merge would be:
> - Alluxio (PR#379)
> - WASB (PR#626)
>  - Azure* (PR#399)
> - *GCP (recently merged as PR#1482)
> - *S3 (although this has been in code so it would have to be deprecated)
> The pattern would be pretty much the same as the one documented and
> successfully deployed here:
> to-azure-data-lake-from-a-nifi-dataflow.html
> Which means that in the case of Alluxio, one would use the properties
> documented here:
> Hadoop-MapReduce-on-Alluxio.html
> While with Google Cloud Storage we would use the properties documented
> here:
> I noticed that specific processors could have the ability to handle
> particular properties to a filesystem, however I would like to believe the
> same issue would plague hadoop users, and therefore is reasonable to
> believe the Hadoop compatible implementations would have ways of exposing
> those properties as well?
> In the case the properties are exposed, we perhaps simply adjust the *HDFS
> processors to use dynamic properties to pass those to the underlying
> module, therefore providing a way to explore particular settings of an
> underlying storage platforms.
> Any opinion would be welcome
> PS-sent it again with proper subject label

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message