nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy LoPresto <>
Subject Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases
Date Wed, 10 Jul 2019 20:16:56 GMT
Thanks Kevin, this looks really promising. 

Updating the link here as I think the page may have moved:

Andy LoPresto
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Jul 10, 2019, at 12:08 PM, Kevin Doran <> wrote:
> Hi NiFi Dev Community,
> Jeff Storck, Bryan Bende, and I have been collaborating back and forth
> on a proposal for how to restructure the NiFi source code into smaller
> Maven projects and repositories based on the discussion that took
> place awhile back on this thread. I'm reviving this older thread in
> order to share that proposal with the community and generate farther
> discussion about at solidifying a destination and a plan for how to
> get there.
> Specifically, the proposal we've started working on has three parts:
> 1. Goals (more or less a summary of the earlier discussion that took
> place on this thread)
> 2. Proposed end state of the new Maven project and repository structure
> 3. Proposed approach for how to get from where we are today to the
> desired end state
> The proposal is on the Apache NiFi Wiki [1], so that we can all
> collaborate on it or leave comments there.
> [1]
> Thanks,
> Kevin, Jeff, and Bryan
> On Thu, May 30, 2019 at 1:31 PM Kevin Doran <> wrote:
>> I am also in favor of splitting the nifi maven project up into smaller
>> projects with independent release cycles in order to decouple
>> development at well defined boundaries/interfaces and also to
>> facilitate code reuse.
>> In anticipation of eventually working towards a NiFi 2.0 that
>> introduces bigger changes for developers and users, I've started work
>> on a nifi-commons project in which I've extracted out some of the code
>> that originally got ported from NiFi -> NiFi Registry, and now exists
>> as similar code in both projects, into a standalone modular library.
>> That premilinary work is here on my personal github account for now
>> [1].
>> So far, it only contains some security code in a submodule, and is a
>> WIP (more work coming when I have time), but the idea is nifi-commons
>> could have several libraries/modules and would be released
>> periodically to use across nifi and registry. If we are talking about
>> spliting the nifi project into framework and extensions, then
>> nifi-commons might be a good home for code that needs to be shared
>> across those two sub projects as well, such as the nifi-api bits Joe
>> mentioned.
>> As part of this larger effort, I would be happy to help get a
>> nifi-commons repository started in Apache where we can move shared
>> code such as nifi-api to prepare for splitting nifi-framework and
>> nifi-extensions. It also occurs to me that if nifi-framework and
>> nifi-extensions are being released independently, nifi-assembly should
>> probably just become a project that pulls in and assembles the latest
>> releases of framework and extensions.
>> Overall, I think this would be beneficial for most of the work going
>> on in Apache NiFi, which would not have to cut across these different
>> project and therefore would be easier to code, test, build, and
>> release. However, the level of difficulty will increase for changes
>> that will need to span multiple projects, though those are fewer in
>> number, so overall I think it would be a net win for the dev
>> community.
>> [1]
>> Thanks,
>> Kevin
>> On Thu, May 30, 2019 at 12:17 PM Andy LoPresto <> wrote:
>>> I am a strong +1 on the separation and reducing the build time. With that in
mind, I think the process I brought up yesterday [1] of signing our artifacts with GPG as
part of the Maven build is paramount, because we would now be consuming core code across multiple
projects/repositories, so there is even less guarantee the code is coming from “us”.
>>> [1]
>>> Andy LoPresto
>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>>> On May 30, 2019, at 9:15 AM, Brandon DeVries <> wrote:
>>>> In regards to "We 'could' also split out the 'nifi-api'...", NiFi 2.0 would
>>>> also be a good time to look at more clearly defining the separation between
>>>> the UI and the framework.  Where nifi-api is the contract between the
>>>> extensions and the framework, the NiFi Rest api is the contract between the
>>>> UI and framework...  These pieces could potentially be built  / deployed
>>>> updated independently.
>>>> On Thu, May 30, 2019 at 11:39 AM Jeff <> wrote:
>>>>> In the same category of challenges that Peter pointed out, it might be
>>>>> difficult for Travis to build the "framework" and "extensions" projects
>>>>> there are changes in a PR that affect both projects.
>>>>> Is there a good way in Travis to have the workspace/maven repo shared
>>>>> between projects in a single build?
>>>>> It's probably always in the direction of the extensions project needing
>>>>> something new to be added to the framework project rather than the other
>>>>> way around, but it'll be tricky to get that working right in Travis if
>>>>> not possible to set up the Travis build to know it needs to deploy the
>>>>> framework project artifacts into a maven repo that the extension project
>>>>> will use.
>>>>> One way might be to make sure that changes to the framework project must
>>>>> in master before the extensions project can make use of them, but that
>>>>> would require a "default master" build for the framework project which
>>>>> builds master after each commit, and deploys the build artifacts to a
>>>>> persistent maven repo that the extension project builds can access. 
>>>>> also makes project-spanning change-sets take longer to review and get
>>>>> committed to master.
>>>>> On Thu, May 30, 2019 at 11:23 AM Peter Wicks (pwicks) <>
>>>>> wrote:
>>>>>> One more "not awesome" would be that core changes that affect extensions
>>>>>> will be a little harder to test. If I make a core change that changes
>>>>>> signature of an interface/etc... I'll need to do some extra work
to make
>>>>>> sure I don't break extensions that use it.
>>>>>> Still worth it, just one more thing to mention.
>>>>>> -----Original Message-----
>>>>>> From: Joe Witt <>
>>>>>> Sent: Thursday, May 30, 2019 9:19 AM
>>>>>> To:
>>>>>> Subject: [EXT] [discuss] Splitting NiFi framework and extension repos
>>>>>> releases
>>>>>> Team,
>>>>>> We've discussed this a bit over the years in various forms but it
>>>>>> seems time to progress this topic and enough has changed I think
>>>>> warrant
>>>>>> it.
>>>>>> Tensions:
>>>>>> 1) Our build times take too long.  In travis-ci for instance it takes
>>>>>> minutes when it works.
>>>>>> 2) The number of builds we do has increased.  We do us/jp/fr builds
>>>>>> open and oracle JDKs.  That is 6 builds.
>>>>>> 3) We want to add Java 11 support such that one could build with
8 or 11
>>>>>> and the above still apply.  The becomes 6 builds.
>>>>>> 4) With the progress in NiFi registry we can now load artifacts there
>>>>>> could pull them into NiFi.  And this integration will only get better.
>>>>>> 5) The NiFi build is too huge and cannot grow any longer or else
>>>>> cannot
>>>>>> upload convenience binaries.
>>>>>> We cannot solve all the things just yet but we can make progress.
>>>>>> suggest we split apart the NiFi 'framework/application' in its own
>>>>> release
>>>>>> cycle and code repository from the 'nifi extensions' into its own
>>>>>> repository and release cycle.  The NiFi release would still pull
in a
>>>>>> specific set of extension bundles so to our end users at this time
>>>>> is
>>>>>> no change. In the future we could also just stop including the extensions
>>>>>> in nifi the application and they could be sourced at runtime as needed
>>>>> from
>>>>>> the registry (call that a NiFi 2.x thing).
>>>>>> Why does this help?
>>>>>> - Builds would only take as long as just extensions take or just
>>>>>> takes.  This reduces time for each change cycle and reduces load
>>>>>> travis-ci which runs the same tests over and over and over for each
>>>>>> request/push regardless of whether it was an extension or core.
>>>>>> - It moves us toward the direction we're heading anyway whereby
>>>>> extensions
>>>>>> can have their own lifecycle from the framework/app itself.
>>>>>> How is this not awesome:
>>>>>> - Doesn't yet solve for the large builds problem.  I think we'll
>>>>> there
>>>>>> with a NiFi 2.x release which fully leverages nifi-registry for retrieval
>>>>>> of all extensions.
>>>>>> - Adds another 'thing we need to do a release cycle for'.  This is
>>>>>> generally unpleasant but it is paid for once a release cycle and
it does
>>>>>> allow us to release independently for new cool extensions/fixes apart
>>>>> from
>>>>>> the framework itself.
>>>>>> Would be great to hear others thoughts if they too feel it is time
>>>>> make
>>>>>> this happen.
>>>>>> Thanks
>>>>>> Joe

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message