nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Bean <mark.o.b...@gmail.com>
Subject Re: [DISCUSS] Extension Registry
Date Tue, 13 Nov 2018 17:56:48 GMT
I would like to see a "group" capability in the Registry for NAR bundles.
Then, users can create their own customized grouping of relevant NARs.
Managing bundles one at a time will become tedious.

Thanks,
Mark

On Tue, Nov 13, 2018 at 12:48 PM Joe Witt <joe.witt@gmail.com> wrote:

> Sivaprasanna - yes absolutely.  That is the core point I think of
> Bryan's first bullet under the 'NiFi' section.
> On Tue, Nov 13, 2018 at 12:47 PM Sivaprasanna <sivaprasanna246@gmail.com>
> wrote:
> >
> > One quick question. With the extension registry, my understanding is that
> > we would try to reduce the overall NiFi size by offloading certain
> existing
> > NAR bundles to the extension registry. So are we expecting the extension
> > registry to also come up with the ability/mechanism that lets an user to
> > download these bundles ?
> >
> > -
> > Sivaprasanna
> >
> > On Tue, 13 Nov 2018 at 11:07 PM, Joe Witt <joe.witt@gmail.com> wrote:
> >
> > > Bryan
> > >
> > > Very exciting to see this under way!!!  We desperately have to get our
> > > core nifi build size way down and make it far easier for people to
> > > contribute new processors.  We have a long line of extensions that
> > > could be really useful/valuable and this will help unlock that while
> > > improving the user experience tremendously.
> > >
> > > For the doc/concerns noted above we should also add/mention that the
> > > relationship between nars (dependencies between nars for
> > > apis/controller services/parent nars/etc..) we want to have a way to
> > > manage/show that or a user experience for it.  Maybe not a day-1 thing
> > > but important to call out.
> > >
> > > Thanks!
> > > Joe
> > > On Tue, Nov 13, 2018 at 12:22 PM Bryan Bende <bbende@gmail.com> wrote:
> > > >
> > > > All,
> > > >
> > > > We've needed the elusive extension registry for quite some time now,
> > > > and with NiFi Registry I think we are in a good place to make some
> > > > progress in this area.
> > > >
> > > > I've started looking into adding "extension bundles" to NiFi Registry
> > > > as the next type of versioned item, along side the existing versioned
> > > > flows, and I wanted to take a minute to outline how that approach
> > > > could work before getting too far into it.
> > > >
> > > > Also, I'd like to focus this discussion on the design and
> > > > functionality of the extension registry, and not on how the community
> > > > is going to use it. Topics like hosting a central registry, changing
> > > > the build process, restructuring the git repo, releasing NARs, etc,
> > > > are all important topics, but first we need an extension registry
> > > > before we can do any of that :)
> > > >
> > > > Here is a high-level description of what needs to be done...
> > > >
> > > > NiFi Registry
> > > >
> > > > - Add a new type of item called an extension bundle, where each
> bundle
> > > > can contain one ore extensions or APIs
> > > >
> > > > - Support bundles for traditional NiFi (aka NARs) and also bundles
> for
> > > > MiNiFi CPP
> > > >
> > > > - Ability to upload the binary artifact for a bundle and extract the
> > > > metadata about the bundle, and metadata about the extensions
> contained
> > > > in the bundle (more on this later)
> > > >
> > > > - Provide a pluggable storage provider for saving the content of each
> > > > extension bundle so that we can have different implementations like
> > > > local fileysystem, S3, and other object stores
> > > >
> > > > - Provide a REST API for listing and retrieving available bundles,
> > > > integrate this into the registry Java client and NiFi CLI
> > > >
> > > > NAR Maven Plugin
> > > >
> > > > - Generate a descriptor for each component in the NAR which will
> > > > provide information like the description, tags, restricted or not,
> > > > etc.
> > > >
> > > > - These descriptors will be parsed by NiFi Registry when a NAR is
> > > > being uploaded so that NiFi Registry will know about the extensions
> > > > contained with in the NAR
> > > >
> > > > NiFi
> > > >
> > > > - Provide some type of extension manager experience where users can
> > > > search, browse, & install bundles that are available in any of the
> > > > registry clients that have been defined
> > > >
> > > > - Introduce a new security policy to control which users are allowed
> > > > to access the extension manager
> > > >
> > > > - Installing a bundle should load the NAR and make the extensions
> > > > available leveraging the recent work done to auto-load new NARs
> > > >
> > > > - Importing versioned flows from registry should provide an easy way
> > > > to install bundles that are required by the flow, but missing from
> the
> > > > local NiFi instance
> > > >
> > > >
> > > > If anyone has any thoughts or concerns about this approach, please
> let
> > > me know.
> > > >
> > > > Thanks,
> > > >
> > > > Bryan
> > >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message