nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aldrin Piri <aldrinp...@gmail.com>
Subject Re: [DISCUSS] MiNiFi Flow Designer
Date Wed, 28 Feb 2018 12:52:59 GMT
It appears I accidentally left off the actual links.  Here they are in all
their glory:

[1]
https://cwiki.apache.org/confluence/display/MINIFI/MiNiFi+Command+and+Control
[2]
https://github.com/apache/nifi-minifi/blob/master/minifi-docs/src/main/markdown/System_Admin_Guide.md#config-file
[3] https://issues.apache.org/jira/browse/MINIFICPP-36
[4] https://github.com/apache/nifi-minifi/tree/master/minifi-c2
[5]
https://github.com/apache/nifi-minifi/tree/master/minifi-c2#configuration-providers
[6]
https://github.com/apache/nifi-minifi/blob/master/minifi-docs/src/main/markdown/System_Admin_Guide.md#automatic-warm-redeploy
[7]
https://cwiki.apache.org/confluence/display/NIFI/Extension+Repositories+%28aka+Extension+Registry%29+for+Dynamically-loaded+Extensions
[8]
https://cwiki.apache.org/confluence/display/MINIFI/MiNiFi+Command+and+Control#MiNiFiCommandandControl-FlowAuthorshipDetails


On Wed, Feb 28, 2018 at 1:53 AM, Aldrin Piri <aldrinpiri@gmail.com> wrote:

> Hey folks,
>
> With the release of Registry, I’ve been contemplating leveraging it and
> the current codebase of NiFi to facilitate ease of use for MiNiFi flow
> design.  One of the areas I would like to form a concerted effort around is
> that of the Command and Control (C2) functionality originally presented
> when the MiNiFi effort was started and further expounded upon with a
> feature proposal [1].  In that proposal, while the names are dated, we have
> components that fulfill some of the core building blocks toward the overall
> vision of C2.
>
> For those not intimately familiar, MiNiFi is primarily configured via a
> YAML configuration file.  [2]  The initial idea was that flows would be
> simple a source, maybe a transform or two, and then transmission.  The
> notion was that hand creation of one of these files would not be overly
> onerous.  It became apparent from questions, however, that users were doing
> more complex flows and this YAML ranged from tedious to difficult for those
> situations.  This precipitated the best approach currently available,
> utilizing a NiFi instance as a way to generate a template.  This template
> was then exported, and using our provided toolkit to get YAML out that is
> deployable with your instance.  Better, but far from ideal.
>
> A couple approaches came to mind to further narrow this gap of user
> expectation and realities of our configuration.
>
> (1) In our current state, we have heavily relied on convenience of
> component overlap.  Generally speaking, we relied on the notion that NiFi’s
> bundled components are a superset of those found in MiNiFi
> implementations.  We could have continued that, but this would not
> necessarily free the user of the file handling and transformation
> processes.  Additionally, this comes with caveats, among them that given
> the ways in which C++ allows us different accesses, there are some
> interesting processors that work better for that more native approach and
> might not ever justify having an equivalent in Java.
>
> (2) Providing a first class experience in NiFi for MiNiFi flow designing
> making use of the existing UI components as a base and a new design mode
> specifically for providing a clean user experience, letting users operate
> in a different context but a familiar environment.  Certainly preferred,
> but how this fit was not clear.
>
> As a crash course in MiNiFi efforts, there have been some key
> contributions that I believe when more strongly united can generate a very
> complete authoring and flow update process.  Some of these items are:
>
>    - C2 API Protocol specification and implementation in C++ [3]
>    - C2 Server with the notion of serving different flows [4] backed by
>    varying implementations, one of which was via REST to access NiFi Templates
>    directly [5]
>    - Warm Redeploy in MiNiFi Java [6]
>
>
> One of the first questions to come to mind was, “Is within NiFi the right
> place?”
>
> I think there are several reasons that this is reasonable. Seeing some of
> the discussion around the consolidation of common UI components presented
> by Scott Aslan, I think sharing the core of the UI elements between the
> traditional NiFi and MiNiFi flow designs is a logical extension.  While
> those flows in MiNiFi are not real-time changes we are dealing with the
> same design elements in creating directed graphs.  Additionally, while
> MiNiFi instances will likely not ever have the same breadth of extensions
> supported, there is nothing inherently prohibitive of a more complex graph
> comprised of larger numbers of edges and/or vertices.  In my view, the
> mechanics are the same but just being performed in a different contexts.
> In the case of MiNiFi, this is effectively an “offline” context.  By
> "offline," I mean we are not strictly bound to loaded components such as
> that in the current NiFi context consisting of items from unpacked NARs.
>
> How can we minimize the impact to NiFi?
>
> Given that this is a new capability, I don’t know that we necessarily need
> to interfere with the existing APIs and could possibly also make this an
> optional feature.  From this standpoint, we could develop new endpoints for
> this specific context.  Some of this work would rely on external services
> and/or definitions to drive the different context.  For example, without
> NARs acting as the source of available components, something like the C2
> server could provide extensions for MiNiFi instances it is aware of to help
> drive the experience.  This data would be a skeletal view of a component
> definition/interface without any of the inherent code backing it needed.
>
> How do we maximize the utility of these efforts?
>
> With the idea about powering MiNiFi flow design with specific sets of
> components, I could see this extending to even NiFi itself.  Consider the
> use case where something like I wish to make use of additional artifacts
> along the lines of [7] and an extension of the current Registry
> implementation.  As a simple example, with an understanding of what the
> universe of components is available to my system and not just those
> currently loaded, it would be possible for users to design for other
> instances.  With these hooks in place, there are a few permutations of this
> relative concept that enable us to carry out some of the “nice to haves” we
> haven’t quite been able to address.
>
> How does this all come together for MiNiFi?
>
> I have created a work of sheer artistic mastery on the Command and Control
> proposal to illustrate a rough sequence and architecture. [8]  This diagram
> shows from entering MiNiFi flow design through to an instance updating to
> the new flow.
>
>
> Overall, Registry was a tremendous step forward for the NiFi ecosystem.  I
> think in our current state, we have reached a convergence of capabilities
> that allow us to build higher level abstractions for dataflow management.
> I think the approach of (2) is a great first foray into that realm that
> provides an immediate solution for a gap in our MiNiFi efforts.
> Additionally, this approach lays the foundation for additional applications
> possibly inclusive of core NiFi.
>
> I look forward to your thoughts and commentary.  Certainly many details to
> think and reason about, but I see some incredible potential with the work
> we've done thus far both near and long term.
>
> Thanks for your time!
> --aldrin
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message