nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy LoPresto <alopre...@apache.org>
Subject Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases
Date Wed, 10 Jul 2019 20:16:56 GMT
Thanks Kevin, this looks really promising. 

Updating the link here as I think the page may have moved: https://cwiki.apache.org/confluence/display/NIFI/NiFi+Project+and+Repository+Restructuring
<https://cwiki.apache.org/confluence/display/NIFI/NiFi+Project+and+Repository+Restructuring>

Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Jul 10, 2019, at 12:08 PM, Kevin Doran <kdoran@apache.org> wrote:
> 
> Hi NiFi Dev Community,
> 
> Jeff Storck, Bryan Bende, and I have been collaborating back and forth
> on a proposal for how to restructure the NiFi source code into smaller
> Maven projects and repositories based on the discussion that took
> place awhile back on this thread. I'm reviving this older thread in
> order to share that proposal with the community and generate farther
> discussion about at solidifying a destination and a plan for how to
> get there.
> 
> Specifically, the proposal we've started working on has three parts:
> 
> 1. Goals (more or less a summary of the earlier discussion that took
> place on this thread)
> 2. Proposed end state of the new Maven project and repository structure
> 3. Proposed approach for how to get from where we are today to the
> desired end state
> 
> The proposal is on the Apache NiFi Wiki [1], so that we can all
> collaborate on it or leave comments there.
> 
> [1] https://cwiki.apache.org/confluence/display/NIFIREG/NiFi+Project+and+Repository+Restructuring
> 
> Thanks,
> Kevin, Jeff, and Bryan
> 
> On Thu, May 30, 2019 at 1:31 PM Kevin Doran <kdoran@apache.org> wrote:
>> 
>> I am also in favor of splitting the nifi maven project up into smaller
>> projects with independent release cycles in order to decouple
>> development at well defined boundaries/interfaces and also to
>> facilitate code reuse.
>> 
>> In anticipation of eventually working towards a NiFi 2.0 that
>> introduces bigger changes for developers and users, I've started work
>> on a nifi-commons project in which I've extracted out some of the code
>> that originally got ported from NiFi -> NiFi Registry, and now exists
>> as similar code in both projects, into a standalone modular library.
>> That premilinary work is here on my personal github account for now
>> [1].
>> 
>> So far, it only contains some security code in a submodule, and is a
>> WIP (more work coming when I have time), but the idea is nifi-commons
>> could have several libraries/modules and would be released
>> periodically to use across nifi and registry. If we are talking about
>> spliting the nifi project into framework and extensions, then
>> nifi-commons might be a good home for code that needs to be shared
>> across those two sub projects as well, such as the nifi-api bits Joe
>> mentioned.
>> 
>> As part of this larger effort, I would be happy to help get a
>> nifi-commons repository started in Apache where we can move shared
>> code such as nifi-api to prepare for splitting nifi-framework and
>> nifi-extensions. It also occurs to me that if nifi-framework and
>> nifi-extensions are being released independently, nifi-assembly should
>> probably just become a project that pulls in and assembles the latest
>> releases of framework and extensions.
>> 
>> Overall, I think this would be beneficial for most of the work going
>> on in Apache NiFi, which would not have to cut across these different
>> project and therefore would be easier to code, test, build, and
>> release. However, the level of difficulty will increase for changes
>> that will need to span multiple projects, though those are fewer in
>> number, so overall I think it would be a net win for the dev
>> community.
>> 
>> [1] https://github.com/kevdoran/nifi-commons
>> 
>> Thanks,
>> Kevin
>> 
>> On Thu, May 30, 2019 at 12:17 PM Andy LoPresto <alopresto@apache.org> wrote:
>>> 
>>> I am a strong +1 on the separation and reducing the build time. With that in
mind, I think the process I brought up yesterday [1] of signing our artifacts with GPG as
part of the Maven build is paramount, because we would now be consuming core code across multiple
projects/repositories, so there is even less guarantee the code is coming from “us”.
>>> 
>>> [1] https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E
<https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E>
>>> 
>>> Andy LoPresto
>>> alopresto@apache.org
>>> alopresto.apache@gmail.com
>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>> 
>>>> On May 30, 2019, at 9:15 AM, Brandon DeVries <brd@jhu.edu> wrote:
>>>> 
>>>> In regards to "We 'could' also split out the 'nifi-api'...", NiFi 2.0 would
>>>> also be a good time to look at more clearly defining the separation between
>>>> the UI and the framework.  Where nifi-api is the contract between the
>>>> extensions and the framework, the NiFi Rest api is the contract between the
>>>> UI and framework...  These pieces could potentially be built  / deployed
/
>>>> updated independently.
>>>> 
>>>> On Thu, May 30, 2019 at 11:39 AM Jeff <jtswork@gmail.com> wrote:
>>>> 
>>>>> In the same category of challenges that Peter pointed out, it might be
>>>>> difficult for Travis to build the "framework" and "extensions" projects
if
>>>>> there are changes in a PR that affect both projects.
>>>>> 
>>>>> Is there a good way in Travis to have the workspace/maven repo shared
>>>>> between projects in a single build?
>>>>> 
>>>>> It's probably always in the direction of the extensions project needing
>>>>> something new to be added to the framework project rather than the other
>>>>> way around, but it'll be tricky to get that working right in Travis if
it's
>>>>> not possible to set up the Travis build to know it needs to deploy the
>>>>> framework project artifacts into a maven repo that the extension project
>>>>> will use.
>>>>> 
>>>>> One way might be to make sure that changes to the framework project must
be
>>>>> in master before the extensions project can make use of them, but that
>>>>> would require a "default master" build for the framework project which
>>>>> builds master after each commit, and deploys the build artifacts to a
>>>>> persistent maven repo that the extension project builds can access. 
It
>>>>> also makes project-spanning change-sets take longer to review and get
fully
>>>>> committed to master.
>>>>> 
>>>>> On Thu, May 30, 2019 at 11:23 AM Peter Wicks (pwicks) <pwicks@micron.com>
>>>>> wrote:
>>>>> 
>>>>>> One more "not awesome" would be that core changes that affect extensions
>>>>>> will be a little harder to test. If I make a core change that changes
the
>>>>>> signature of an interface/etc... I'll need to do some extra work
to make
>>>>>> sure I don't break extensions that use it.
>>>>>> 
>>>>>> Still worth it, just one more thing to mention.
>>>>>> 
>>>>>> -----Original Message-----
>>>>>> From: Joe Witt <joewitt@apache.org>
>>>>>> Sent: Thursday, May 30, 2019 9:19 AM
>>>>>> To: dev@nifi.apache.org
>>>>>> Subject: [EXT] [discuss] Splitting NiFi framework and extension repos
and
>>>>>> releases
>>>>>> 
>>>>>> Team,
>>>>>> 
>>>>>> We've discussed this a bit over the years in various forms but it
again
>>>>>> seems time to progress this topic and enough has changed I think
to
>>>>> warrant
>>>>>> it.
>>>>>> 
>>>>>> Tensions:
>>>>>> 1) Our build times take too long.  In travis-ci for instance it takes
40
>>>>>> minutes when it works.
>>>>>> 2) The number of builds we do has increased.  We do us/jp/fr builds
on
>>>>>> open and oracle JDKs.  That is 6 builds.
>>>>>> 3) We want to add Java 11 support such that one could build with
8 or 11
>>>>>> and the above still apply.  The becomes 6 builds.
>>>>>> 4) With the progress in NiFi registry we can now load artifacts there
and
>>>>>> could pull them into NiFi.  And this integration will only get better.
>>>>>> 5) The NiFi build is too huge and cannot grow any longer or else
we
>>>>> cannot
>>>>>> upload convenience binaries.
>>>>>> 
>>>>>> We cannot solve all the things just yet but we can make progress.
 I
>>>>>> suggest we split apart the NiFi 'framework/application' in its own
>>>>> release
>>>>>> cycle and code repository from the 'nifi extensions' into its own
>>>>>> repository and release cycle.  The NiFi release would still pull
in a
>>>>>> specific set of extension bundles so to our end users at this time
there
>>>>> is
>>>>>> no change. In the future we could also just stop including the extensions
>>>>>> in nifi the application and they could be sourced at runtime as needed
>>>>> from
>>>>>> the registry (call that a NiFi 2.x thing).
>>>>>> 
>>>>>> Why does this help?
>>>>>> - Builds would only take as long as just extensions take or just
core/app
>>>>>> takes.  This reduces time for each change cycle and reduces load
on
>>>>>> travis-ci which runs the same tests over and over and over for each
pull
>>>>>> request/push regardless of whether it was an extension or core.
>>>>>> 
>>>>>> - It moves us toward the direction we're heading anyway whereby
>>>>> extensions
>>>>>> can have their own lifecycle from the framework/app itself.
>>>>>> 
>>>>>> How is this not awesome:
>>>>>> - Doesn't yet solve for the large builds problem.  I think we'll
get
>>>>> there
>>>>>> with a NiFi 2.x release which fully leverages nifi-registry for retrieval
>>>>>> of all extensions.
>>>>>> - Adds another 'thing we need to do a release cycle for'.  This is
>>>>>> generally unpleasant but it is paid for once a release cycle and
it does
>>>>>> allow us to release independently for new cool extensions/fixes apart
>>>>> from
>>>>>> the framework itself.
>>>>>> 
>>>>>> Would be great to hear others thoughts if they too feel it is time
to
>>>>> make
>>>>>> this happen.
>>>>>> 
>>>>>> Thanks
>>>>>> Joe
>>>>>> 
>>>>> 
>>> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message