incubator-nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edenfield, Orrin" <orrin.edenfi...@prgx.com>
Subject Multiple dataflows with sub-flows and version control
Date Fri, 02 Jan 2015 16:36:42 GMT
Hello everyone - I'm new to the mailing list and I've tried to search the JIRA and mailing
list to see if this has already been addressed and didn't find anything so here it goes:

When I think about the capabilities of this tool I instantly think of ETL-type tools. So the
questions/comments below are likely to be coming from that frame of mind - let me know if
I've misunderstood a key concept of NiFi as I think that could be possible.

Is it possible to have NiFi service setup and running and allow for multiple dataflows to
be designed and deployed (running) at the same time?  So far in my testing I've found that
I can get NiFi service up and functioning as expected on my cluster edge node but I'd like
to be able to design multiple dataflows for the following reasons.

1. I have many datasets that will need some of the same flow actions but not all of them.
I'd like to componentize the flows and possibly have multiple flows cascade from one to another.
For example:  I will want all data to flow into an HDFS endpoint but dataset1 will be coming
in as delimited data so it can go directly into the GetFile processor while I need dataset2
to go through a CompressContent processor first.

2. Because I have a need in #1 above - I'd like to be able to design multiple flows (specific
to a data need or component flows that work together) and have them all be able to be deployed
(running) concurrently.

Also - it would be nice to be able to version control these designed flows so I can have 1
flow running while modifying a version 2.0 of that flow and then once the updates have been
made then I can safely and effectively have a mechanism to shut down flow.v1 and start up
flow.v2.

Thank you.

--
Orrin Edenfield

Mime
View raw message