nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aldrin Piri <aldrinp...@gmail.com>
Subject MiNiFi Command and Control, NiFi SDLC and Flow Versioning
Date Sun, 02 Oct 2016 21:09:10 GMT
Hey folks,



I have been thinking about establishing a game plan to start pursuing some
of the tenets of MiNiFi Command and Control and captured these in a new
Feature Proposal on the MiNiFi wiki[1].  With this page, I have created
some initial design documentation to aid in articulating some of the cases
and functionality we are looking to provide predicated on the discussions
surrounding MiNiFi's inception.


The initial thrust was that of command and control of MiNiFi instances and
standing up additional infrastructure to establish bi-directional
communications.  There have been some initial community discussions as to
what this would look like and the interactions that were expected.  For the
first effort, accomplishing the ability to have a mechanism to provide
updated flows to allow instances to roll over to new flows would be a great
value to aid in the management of dataflow.  A simpler goal, but comes with
a lot of nuance to consider.


As is the case with systems detached from core infrastructure and
networking, connectivity issues are a concern.  When it comes to handling
data that is collected and the processing after it has reached its
destination, comprehension of the data and its path to this point are
important.  Provenance and attributes can play a big part of this, but as
versions of processing flows may diverge being able to understand what is
still of value is critical.  It is with this, that the capability of being
able to version flows is also of great importance.


As a result, I have also added some additional context and background to
the initial feature proposal of Configuration Management of Flows [2] to
help support the tagging of MiNiFi data.  Flow versioning is but one
component of this overall CM process and NiFi user SDLC that is important
for being able to reason about the data that is received from MiNiFi
instances.  However, there are many shared benefits that NiFi would also be
able to utilize for various needs such as migration of flows between
varying levels of production and creating richer context of provenance
events in the face of evolving flows and processing.


I would appreciate anyone interested in things MiNiFi C&C or has sought
better ways to manage NiFi flows between environments to comment on some of
the initial content at the linked proposals.  Feedback or ideas to help
guide the direction of these efforts will aid greatly as we move forward on
establishing design.  These proposals are likely to introduce new
components and interfaces into what is evolving into a NiFi ecosystem of
sorts.  Being mindful of the use cases that the community has for some of
the functionalities that support these goals would allow a more singular
approach of the overall management of dataflow despite where it exists in
its lifecycle.  With features such as these getting incorporated with some
of the other components mentioned, we will have the capabilities to do some
really exciting and impressive management of data.


Thanks!

--aldrin


[1]
https://cwiki.apache.org/confluence/display/MINIFI/MiNiFi+Command+and+Control

[2]
https://cwiki.apache.org/confluence/display/NIFI/Configuration+Management+of+Flows#ConfigurationManagementofFlows-FlowVersioning

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message