From dev-return-18481-archive-asf-public=cust-asf.ponee.io@nifi.apache.org Tue Nov 13 19:00:37 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 6C9DF18062B for ; Tue, 13 Nov 2018 19:00:36 +0100 (CET) Received: (qmail 5855 invoked by uid 500); 13 Nov 2018 18:00:35 -0000 Mailing-List: contact dev-help@nifi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@nifi.apache.org Delivered-To: mailing list dev@nifi.apache.org Received: (qmail 5843 invoked by uid 99); 13 Nov 2018 18:00:34 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Nov 2018 18:00:34 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 4F0CFC1BBD for ; Tue, 13 Nov 2018 18:00:34 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -1.66 X-Spam-Level: X-Spam-Status: No, score=-1.66 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-1.459, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id q8FArDs2ovH9 for ; Tue, 13 Nov 2018 18:00:32 +0000 (UTC) Received: from mail-vk1-f175.google.com (mail-vk1-f175.google.com [209.85.221.175]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 15E415FE63 for ; Tue, 13 Nov 2018 18:00:32 +0000 (UTC) Received: by mail-vk1-f175.google.com with SMTP id j23so2838426vke.13 for ; Tue, 13 Nov 2018 10:00:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=8W/d7JcMsddCGDucIUIHW8qg1QyqhyL/wJUmidb/Hfk=; b=UjIqIvVi/bDYo8Aofah9kNvEtzjGmBjN+G6IFgVcghxsSQmBzluG8gW3Ft7Rb80TxR qq9B0TYhp5mdeu6nikzAn/hyboO+S/AeRk5/zEc3QwmfxPGKMmFQOIP/XciI6oTW4wCH 20oJibT6tgqcecHsL1InK3smfHDo3yzw9l3LCg/BKTv+3/uTxSS7GpFRlaOLpnPBjxjA zE8B8yXOuK+kbKZq2MjY3w8g1cUz2e2JMubA3iIM+mb88oSmAvxJCCcTJj45DLM0SIYN wLdGSXerHlb3Lvbfq8Tt1wY1MM7wCTX3dBiTRFXj3pLARIXiYF8BI8ZKH1kJNqcTBEpK b3eg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=8W/d7JcMsddCGDucIUIHW8qg1QyqhyL/wJUmidb/Hfk=; b=EI8k6YzIRoFYnTXJjvP/kqAjsMP1eCLnwbOYMI+/ZMagrQyI30SqSDfsI3b71SoTak jsfLxSBO6VywAXJDP3q8+hI3v2E/TkY2JJh5WHQj4ZNzPxy1NrJhYWfWkw2pMIgedVkz VKAq166o1b1eJwfjySDKej+0SMUsAbgAgUk/aEyTbxIFWKmROD9FZyUBDu3Mjkb7fx+p OwHrN2zit5YEWIg6ww/565GzO+R3Yg38OPBmljF7UPTrEO7p2+0sIDIFbO+OgR9fKMyt tV6HrgfO+FClLCt8jfiRlVXDrUl10IGUIyubVivBFjO6tYsIKSY9j11rlqPQIIEdYeDS uKQg== X-Gm-Message-State: AGRZ1gKxrvaTQ0vQh+aBxXqgeroi4cv/En1rORSS8/RahYKzCc0HJ6oX C4IUaxSZNp6lGa4EMElDsp1sbKUQt97airJWQ1rtfg== X-Google-Smtp-Source: AJdET5cXvRroNK2SD2AXzobpOhFt4p3vUa72RBwgKPgTJgAIP9gjjobWw3qaa3+wTsHWJxy5qnFLphN/pWWAoD53Aeo= X-Received: by 2002:a1f:5cc7:: with SMTP id q190mr2653077vkb.91.1542132030332; Tue, 13 Nov 2018 10:00:30 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Joe Witt Date: Tue, 13 Nov 2018 13:00:18 -0500 Message-ID: Subject: Re: [DISCUSS] Extension Registry To: dev@nifi.apache.org Content-Type: text/plain; charset="UTF-8" Mark Can you describe your use case from the user perspective both for the entity that would upload the items and demarcate them as a group as well as the user that would consume those bundles? I ask because the point here is that nars are themselves a 'group' in that they are a logical/contained grouping of extensions. These can have relationships to other nars as we know. And flows are designed against specific components that come from certain nars for which we know the precise coordinates. When importing flows that depend on these the system will be able to automatically acquire all that it needs. Thanks On Tue, Nov 13, 2018 at 12:57 PM Mark Bean wrote: > > I would like to see a "group" capability in the Registry for NAR bundles. > Then, users can create their own customized grouping of relevant NARs. > Managing bundles one at a time will become tedious. > > Thanks, > Mark > > On Tue, Nov 13, 2018 at 12:48 PM Joe Witt wrote: > > > Sivaprasanna - yes absolutely. That is the core point I think of > > Bryan's first bullet under the 'NiFi' section. > > On Tue, Nov 13, 2018 at 12:47 PM Sivaprasanna > > wrote: > > > > > > One quick question. With the extension registry, my understanding is that > > > we would try to reduce the overall NiFi size by offloading certain > > existing > > > NAR bundles to the extension registry. So are we expecting the extension > > > registry to also come up with the ability/mechanism that lets an user to > > > download these bundles ? > > > > > > - > > > Sivaprasanna > > > > > > On Tue, 13 Nov 2018 at 11:07 PM, Joe Witt wrote: > > > > > > > Bryan > > > > > > > > Very exciting to see this under way!!! We desperately have to get our > > > > core nifi build size way down and make it far easier for people to > > > > contribute new processors. We have a long line of extensions that > > > > could be really useful/valuable and this will help unlock that while > > > > improving the user experience tremendously. > > > > > > > > For the doc/concerns noted above we should also add/mention that the > > > > relationship between nars (dependencies between nars for > > > > apis/controller services/parent nars/etc..) we want to have a way to > > > > manage/show that or a user experience for it. Maybe not a day-1 thing > > > > but important to call out. > > > > > > > > Thanks! > > > > Joe > > > > On Tue, Nov 13, 2018 at 12:22 PM Bryan Bende wrote: > > > > > > > > > > All, > > > > > > > > > > We've needed the elusive extension registry for quite some time now, > > > > > and with NiFi Registry I think we are in a good place to make some > > > > > progress in this area. > > > > > > > > > > I've started looking into adding "extension bundles" to NiFi Registry > > > > > as the next type of versioned item, along side the existing versioned > > > > > flows, and I wanted to take a minute to outline how that approach > > > > > could work before getting too far into it. > > > > > > > > > > Also, I'd like to focus this discussion on the design and > > > > > functionality of the extension registry, and not on how the community > > > > > is going to use it. Topics like hosting a central registry, changing > > > > > the build process, restructuring the git repo, releasing NARs, etc, > > > > > are all important topics, but first we need an extension registry > > > > > before we can do any of that :) > > > > > > > > > > Here is a high-level description of what needs to be done... > > > > > > > > > > NiFi Registry > > > > > > > > > > - Add a new type of item called an extension bundle, where each > > bundle > > > > > can contain one ore extensions or APIs > > > > > > > > > > - Support bundles for traditional NiFi (aka NARs) and also bundles > > for > > > > > MiNiFi CPP > > > > > > > > > > - Ability to upload the binary artifact for a bundle and extract the > > > > > metadata about the bundle, and metadata about the extensions > > contained > > > > > in the bundle (more on this later) > > > > > > > > > > - Provide a pluggable storage provider for saving the content of each > > > > > extension bundle so that we can have different implementations like > > > > > local fileysystem, S3, and other object stores > > > > > > > > > > - Provide a REST API for listing and retrieving available bundles, > > > > > integrate this into the registry Java client and NiFi CLI > > > > > > > > > > NAR Maven Plugin > > > > > > > > > > - Generate a descriptor for each component in the NAR which will > > > > > provide information like the description, tags, restricted or not, > > > > > etc. > > > > > > > > > > - These descriptors will be parsed by NiFi Registry when a NAR is > > > > > being uploaded so that NiFi Registry will know about the extensions > > > > > contained with in the NAR > > > > > > > > > > NiFi > > > > > > > > > > - Provide some type of extension manager experience where users can > > > > > search, browse, & install bundles that are available in any of the > > > > > registry clients that have been defined > > > > > > > > > > - Introduce a new security policy to control which users are allowed > > > > > to access the extension manager > > > > > > > > > > - Installing a bundle should load the NAR and make the extensions > > > > > available leveraging the recent work done to auto-load new NARs > > > > > > > > > > - Importing versioned flows from registry should provide an easy way > > > > > to install bundles that are required by the flow, but missing from > > the > > > > > local NiFi instance > > > > > > > > > > > > > > > If anyone has any thoughts or concerns about this approach, please > > let > > > > me know. > > > > > > > > > > Thanks, > > > > > > > > > > Bryan > > > > > >