From dev-return-16545-archive-asf-public=cust-asf.ponee.io@nifi.apache.org Wed Feb 28 13:53:46 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 78013180657 for ; Wed, 28 Feb 2018 13:53:45 +0100 (CET) Received: (qmail 31255 invoked by uid 500); 28 Feb 2018 12:53:44 -0000 Mailing-List: contact dev-help@nifi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@nifi.apache.org Delivered-To: mailing list dev@nifi.apache.org Received: (qmail 31237 invoked by uid 99); 28 Feb 2018 12:53:43 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Feb 2018 12:53:43 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 2D2D8C1F8A for ; Wed, 28 Feb 2018 12:53:43 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.879 X-Spam-Level: * X-Spam-Status: No, score=1.879 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id 3pKlzUSnWWNe for ; Wed, 28 Feb 2018 12:53:40 +0000 (UTC) Received: from mail-it0-f49.google.com (mail-it0-f49.google.com [209.85.214.49]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 85CF05F173 for ; Wed, 28 Feb 2018 12:53:40 +0000 (UTC) Received: by mail-it0-f49.google.com with SMTP id k79so3382885ita.2 for ; Wed, 28 Feb 2018 04:53:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=I1BpK/3N1STTTgCFtsrP81DAlqkkOfHrhcLAqNHK32I=; b=ttOV+4BOmKdyGKki2FyOaTV+yb1cOVu9xLVFBc9zW9GnbJUs0K9980SXYw6+sg2tfj 65L7WNN/KozjSfSNYdPGw5SleBJLkcSe+0bTnoQdmcQn6ryylivu0jQoEHEGMqvbCL7T KPZXqWEerNFOCtVWJ/oCxtQ9mWdpmjiA/eDzTJdmD47c8bLNlmKZY/5WM00pG+JrLGrF vL7ZgOdlzvkJiKwkyUSGTr6V0v2xAZJ3VI31FEerMuqarJu2rm9LsU9/FcCBuMMpJBYv EDeJ6fwE1ghVQJ0d2nlIVyCuNoFS58x+ONbL+an+kC3krBhi8j/CBtX7Qt5u+S174m2g kYbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=I1BpK/3N1STTTgCFtsrP81DAlqkkOfHrhcLAqNHK32I=; b=padwAsxLWN8BrbER4azn5JnwJ0wNF5fEUxua/TW7vw2BM3TUnkJ9414tPR9gS0BnZq jV1+dJSHX/cIl5rCxNyJ5jtawQ50Ln+Rj/S5WuYxOGtVNbGkh59JKBuZF2ifP9YuBmWG T0YVuEbkB+v48rpA8ugY1LBRaRwDApJu4Z+t52jci2ZpBbK4mlNlkZ9cRUqIt8ytS89R ACItvC3jyeRgIJkuPtwYIhMWpWepzui+gY9WWlydmAdTuDG5WR2tNNAaY9nIu85Jv4wW V49lS2AfrmKeVx85atgu4mBJfE+xonhe53rRWPDr9PeSLIXo3BzEyuhNpSrepCp5x9XL 7vWw== X-Gm-Message-State: APf1xPAKJrZ2uEkvaIUS2iQZG/IP0onGHmfaZJka23Aj37hISfymjQfo q8EHbwqBEZf6Oo6KpdEK3sKuXERC/rS5btPLEHoPds3L X-Google-Smtp-Source: AG47ELvR9/iHfQqXRSZUtSRSI7Ay0ZnAfWfed/tK5khrDL0I5lGwI5bAk25+aOZGk5Th7uHqpyC+oNoMne8KLXNC81w= X-Received: by 10.36.1.149 with SMTP id 143mr21052176itk.112.1519822419585; Wed, 28 Feb 2018 04:53:39 -0800 (PST) MIME-Version: 1.0 Received: by 10.2.164.11 with HTTP; Wed, 28 Feb 2018 04:52:59 -0800 (PST) In-Reply-To: References: From: Aldrin Piri Date: Wed, 28 Feb 2018 07:52:59 -0500 Message-ID: Subject: Re: [DISCUSS] MiNiFi Flow Designer To: dev Content-Type: multipart/alternative; boundary="001a1143ce6070f0020566453be1" --001a1143ce6070f0020566453be1 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable It appears I accidentally left off the actual links. Here they are in all their glory: [1] https://cwiki.apache.org/confluence/display/MINIFI/MiNiFi+Command+and+Contr= ol [2] https://github.com/apache/nifi-minifi/blob/master/minifi-docs/src/main/mark= down/System_Admin_Guide.md#config-file [3] https://issues.apache.org/jira/browse/MINIFICPP-36 [4] https://github.com/apache/nifi-minifi/tree/master/minifi-c2 [5] https://github.com/apache/nifi-minifi/tree/master/minifi-c2#configuration-p= roviders [6] https://github.com/apache/nifi-minifi/blob/master/minifi-docs/src/main/mark= down/System_Admin_Guide.md#automatic-warm-redeploy [7] https://cwiki.apache.org/confluence/display/NIFI/Extension+Repositories+%28= aka+Extension+Registry%29+for+Dynamically-loaded+Extensions [8] https://cwiki.apache.org/confluence/display/MINIFI/MiNiFi+Command+and+Contr= ol#MiNiFiCommandandControl-FlowAuthorshipDetails On Wed, Feb 28, 2018 at 1:53 AM, Aldrin Piri wrote: > Hey folks, > > With the release of Registry, I=E2=80=99ve been contemplating leveraging = it and > the current codebase of NiFi to facilitate ease of use for MiNiFi flow > design. One of the areas I would like to form a concerted effort around = is > that of the Command and Control (C2) functionality originally presented > when the MiNiFi effort was started and further expounded upon with a > feature proposal [1]. In that proposal, while the names are dated, we ha= ve > components that fulfill some of the core building blocks toward the overa= ll > vision of C2. > > For those not intimately familiar, MiNiFi is primarily configured via a > YAML configuration file. [2] The initial idea was that flows would be > simple a source, maybe a transform or two, and then transmission. The > notion was that hand creation of one of these files would not be overly > onerous. It became apparent from questions, however, that users were doi= ng > more complex flows and this YAML ranged from tedious to difficult for tho= se > situations. This precipitated the best approach currently available, > utilizing a NiFi instance as a way to generate a template. This template > was then exported, and using our provided toolkit to get YAML out that is > deployable with your instance. Better, but far from ideal. > > A couple approaches came to mind to further narrow this gap of user > expectation and realities of our configuration. > > (1) In our current state, we have heavily relied on convenience of > component overlap. Generally speaking, we relied on the notion that NiFi= =E2=80=99s > bundled components are a superset of those found in MiNiFi > implementations. We could have continued that, but this would not > necessarily free the user of the file handling and transformation > processes. Additionally, this comes with caveats, among them that given > the ways in which C++ allows us different accesses, there are some > interesting processors that work better for that more native approach and > might not ever justify having an equivalent in Java. > > (2) Providing a first class experience in NiFi for MiNiFi flow designing > making use of the existing UI components as a base and a new design mode > specifically for providing a clean user experience, letting users operate > in a different context but a familiar environment. Certainly preferred, > but how this fit was not clear. > > As a crash course in MiNiFi efforts, there have been some key > contributions that I believe when more strongly united can generate a ver= y > complete authoring and flow update process. Some of these items are: > > - C2 API Protocol specification and implementation in C++ [3] > - C2 Server with the notion of serving different flows [4] backed by > varying implementations, one of which was via REST to access NiFi Temp= lates > directly [5] > - Warm Redeploy in MiNiFi Java [6] > > > One of the first questions to come to mind was, =E2=80=9CIs within NiFi t= he right > place?=E2=80=9D > > I think there are several reasons that this is reasonable. Seeing some of > the discussion around the consolidation of common UI components presented > by Scott Aslan, I think sharing the core of the UI elements between the > traditional NiFi and MiNiFi flow designs is a logical extension. While > those flows in MiNiFi are not real-time changes we are dealing with the > same design elements in creating directed graphs. Additionally, while > MiNiFi instances will likely not ever have the same breadth of extensions > supported, there is nothing inherently prohibitive of a more complex grap= h > comprised of larger numbers of edges and/or vertices. In my view, the > mechanics are the same but just being performed in a different contexts. > In the case of MiNiFi, this is effectively an =E2=80=9Coffline=E2=80=9D c= ontext. By > "offline," I mean we are not strictly bound to loaded components such as > that in the current NiFi context consisting of items from unpacked NARs. > > How can we minimize the impact to NiFi? > > Given that this is a new capability, I don=E2=80=99t know that we necessa= rily need > to interfere with the existing APIs and could possibly also make this an > optional feature. From this standpoint, we could develop new endpoints f= or > this specific context. Some of this work would rely on external services > and/or definitions to drive the different context. For example, without > NARs acting as the source of available components, something like the C2 > server could provide extensions for MiNiFi instances it is aware of to he= lp > drive the experience. This data would be a skeletal view of a component > definition/interface without any of the inherent code backing it needed. > > How do we maximize the utility of these efforts? > > With the idea about powering MiNiFi flow design with specific sets of > components, I could see this extending to even NiFi itself. Consider the > use case where something like I wish to make use of additional artifacts > along the lines of [7] and an extension of the current Registry > implementation. As a simple example, with an understanding of what the > universe of components is available to my system and not just those > currently loaded, it would be possible for users to design for other > instances. With these hooks in place, there are a few permutations of th= is > relative concept that enable us to carry out some of the =E2=80=9Cnice to= haves=E2=80=9D we > haven=E2=80=99t quite been able to address. > > How does this all come together for MiNiFi? > > I have created a work of sheer artistic mastery on the Command and Contro= l > proposal to illustrate a rough sequence and architecture. [8] This diagr= am > shows from entering MiNiFi flow design through to an instance updating to > the new flow. > > > Overall, Registry was a tremendous step forward for the NiFi ecosystem. = I > think in our current state, we have reached a convergence of capabilities > that allow us to build higher level abstractions for dataflow management. > I think the approach of (2) is a great first foray into that realm that > provides an immediate solution for a gap in our MiNiFi efforts. > Additionally, this approach lays the foundation for additional applicatio= ns > possibly inclusive of core NiFi. > > I look forward to your thoughts and commentary. Certainly many details t= o > think and reason about, but I see some incredible potential with the work > we've done thus far both near and long term. > > Thanks for your time! > --aldrin > > --001a1143ce6070f0020566453be1--