nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aldrin Piri <aldrinp...@gmail.com>
Subject [DISCUSS] MiNiFi Flow Designer
Date Wed, 28 Feb 2018 06:53:27 GMT
 Hey folks,

With the release of Registry, I’ve been contemplating leveraging it and the
current codebase of NiFi to facilitate ease of use for MiNiFi flow design.
One of the areas I would like to form a concerted effort around is that of
the Command and Control (C2) functionality originally presented when the
MiNiFi effort was started and further expounded upon with a feature
proposal [1].  In that proposal, while the names are dated, we have
components that fulfill some of the core building blocks toward the overall
vision of C2.

For those not intimately familiar, MiNiFi is primarily configured via a
YAML configuration file.  [2]  The initial idea was that flows would be
simple a source, maybe a transform or two, and then transmission.  The
notion was that hand creation of one of these files would not be overly
onerous.  It became apparent from questions, however, that users were doing
more complex flows and this YAML ranged from tedious to difficult for those
situations.  This precipitated the best approach currently available,
utilizing a NiFi instance as a way to generate a template.  This template
was then exported, and using our provided toolkit to get YAML out that is
deployable with your instance.  Better, but far from ideal.

A couple approaches came to mind to further narrow this gap of user
expectation and realities of our configuration.

(1) In our current state, we have heavily relied on convenience of
component overlap.  Generally speaking, we relied on the notion that NiFi’s
bundled components are a superset of those found in MiNiFi
implementations.  We could have continued that, but this would not
necessarily free the user of the file handling and transformation
processes.  Additionally, this comes with caveats, among them that given
the ways in which C++ allows us different accesses, there are some
interesting processors that work better for that more native approach and
might not ever justify having an equivalent in Java.

(2) Providing a first class experience in NiFi for MiNiFi flow designing
making use of the existing UI components as a base and a new design mode
specifically for providing a clean user experience, letting users operate
in a different context but a familiar environment.  Certainly preferred,
but how this fit was not clear.

As a crash course in MiNiFi efforts, there have been some key contributions
that I believe when more strongly united can generate a very complete
authoring and flow update process.  Some of these items are:

   - C2 API Protocol specification and implementation in C++ [3]
   - C2 Server with the notion of serving different flows [4] backed by
   varying implementations, one of which was via REST to access NiFi Templates
   directly [5]
   - Warm Redeploy in MiNiFi Java [6]


One of the first questions to come to mind was, “Is within NiFi the right
place?”

I think there are several reasons that this is reasonable. Seeing some of
the discussion around the consolidation of common UI components presented
by Scott Aslan, I think sharing the core of the UI elements between the
traditional NiFi and MiNiFi flow designs is a logical extension.  While
those flows in MiNiFi are not real-time changes we are dealing with the
same design elements in creating directed graphs.  Additionally, while
MiNiFi instances will likely not ever have the same breadth of extensions
supported, there is nothing inherently prohibitive of a more complex graph
comprised of larger numbers of edges and/or vertices.  In my view, the
mechanics are the same but just being performed in a different contexts.
In the case of MiNiFi, this is effectively an “offline” context.  By
"offline," I mean we are not strictly bound to loaded components such as
that in the current NiFi context consisting of items from unpacked NARs.

How can we minimize the impact to NiFi?

Given that this is a new capability, I don’t know that we necessarily need
to interfere with the existing APIs and could possibly also make this an
optional feature.  From this standpoint, we could develop new endpoints for
this specific context.  Some of this work would rely on external services
and/or definitions to drive the different context.  For example, without
NARs acting as the source of available components, something like the C2
server could provide extensions for MiNiFi instances it is aware of to help
drive the experience.  This data would be a skeletal view of a component
definition/interface without any of the inherent code backing it needed.

How do we maximize the utility of these efforts?

With the idea about powering MiNiFi flow design with specific sets of
components, I could see this extending to even NiFi itself.  Consider the
use case where something like I wish to make use of additional artifacts
along the lines of [7] and an extension of the current Registry
implementation.  As a simple example, with an understanding of what the
universe of components is available to my system and not just those
currently loaded, it would be possible for users to design for other
instances.  With these hooks in place, there are a few permutations of this
relative concept that enable us to carry out some of the “nice to haves” we
haven’t quite been able to address.

How does this all come together for MiNiFi?

I have created a work of sheer artistic mastery on the Command and Control
proposal to illustrate a rough sequence and architecture. [8]  This diagram
shows from entering MiNiFi flow design through to an instance updating to
the new flow.


Overall, Registry was a tremendous step forward for the NiFi ecosystem.  I
think in our current state, we have reached a convergence of capabilities
that allow us to build higher level abstractions for dataflow management.
I think the approach of (2) is a great first foray into that realm that
provides an immediate solution for a gap in our MiNiFi efforts.
Additionally, this approach lays the foundation for additional applications
possibly inclusive of core NiFi.

I look forward to your thoughts and commentary.  Certainly many details to
think and reason about, but I see some incredible potential with the work
we've done thus far both near and long term.

Thanks for your time!
--aldrin

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message