nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevin Doran <kdo...@apache.org>
Subject Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases
Date Fri, 12 Jul 2019 17:05:44 GMT
Thanks Adam and Edward! This is exactly the type of discussion I was
hoping a detailed and specific proposal would generate, so thanks for
the input. I'll reply to each of you in turn:

Adam,

It is true that a repo-per-project approach is not required. I've
worked on projects that do it both ways and there are advantages to
both.

Single-repo was considered, but as one of the primary goals is to cut
down on Travis / CI build times, the mutli-repo approach seemed to
have a big advantage. Personally, I've never found a reliable, stable
way to introduce CI builds to a repository with multiple projects that
did not require building all the projects in the repository. It's
possible to try to use commands to determine which files have changed
and infer which project(s) to build from that, but maintaining that
logic can get messy. If the logic is wrong, it's possible a project
that is not built is broken by a PR. Building everything is not an
option for a project our size, as our build already time out today.
Fast, reliable Travis builds with no false positives / negatives is
definitely something NiFi needs, and I think it will be simplest to
get there with a multi-repo approach.

That said, I agree that the *biggest* win comes from splitting
projects, and that splitting repos is a smaller step. I don't feel
strongly about it and could live with a single repo with multiple
projects (though, for what it's worth, the NiFi umbrella already has
several repositories and I personally don't feel it has been
burdensome).

And I agree - let's not start splitting JIRA projects. Let's use
components or labels or something to differentiate issues under the
existing NIFI Jira project.


Edward,

Thanks. I totally agree and I know others who feel the same way.
Better defined boundaries and loosely coupled modules is 100% a
long-term goal. I think this project restructuring won't solve the
problem completely (in fact, to your point, it may uncover some
unfortunate tight-coupling that needs to be reworked on the current
master before the split can happen), but I do think it will encourage
developers to more faithfully build to APIs and avoid leaky
abstractions as there will be more hard division points in the code
base. Some of those issues might be able to be addressed immediately.
Others might have to wait for a major version change.

Thanks,
Kevin

On Fri, Jul 12, 2019 at 1:04 PM Adam Taft <adam@adamtaft.com> wrote:
>
> To be honest and to your point Joe, the thing that optimizes the RM duties
> should probably be preferred in all of this.  There is so much overhead for
> the release manager, that lubricating the RM process probably trumps a lot
> of my concerns.  I think there's real concern for making the project harder
> for new contributors. But likewise, that concern should be balanced with
> making the project harder for longtime contributors who have pulled the
> cart the most.
>
> I was just at least hoping for a discussion on the concept.  Thanks as
> always for your leadership and contributions to the nifi community.
>
> On Fri, Jul 12, 2019 at 10:48 AM Joe Witt <joe.witt@gmail.com> wrote:
>
> > Ah I agree the JIRA thing would be too heavy handed.  A single JIRA with
> > well defined components tied to 'repos' is good.
> >
> > As far as separate code repos we're talking about different releasable
> > artifacts for which we as a PMC are responsible for the meaning/etc..  As a
> > many time RM I definitely dislike the mono repo construct as I understand
> > it to function.  I prefer repos per source release artifact where all
> > source in that artifact is a function of the release. I am ok with
> > different convenience binaries resulting from a single source release
> > artifact though.
> >
> > Thanks
> >
> > On Fri, Jul 12, 2019 at 12:26 PM Adam Taft <adam@adamtaft.com> wrote:
> >
> > > I think the concerns around user management are valid, are they not?
> > > Overhead in JIRA goes up (assigning rights to users in JIRA is
> > > multiplied).  Risk to new contributors is high, because each isolated
> > > repository has its own life and code contribution styles.  Maybe the
> > actual
> > > apache infra involvement is low, but the negative effects of community
> > and
> > > source code bifurcation goes up.
> > >
> > > Tagging in mono-repos is done by prefixing the name of the component in
> > the
> > > tag name.  Your release sources are still generated from the component
> > > folder (not from the root).
> > >
> > > Modularization (as being proposed) is a good thing, but can be done in a
> > > single repository. It's not a requirement to split up the git project to
> > > get the benefits of modularization.  That's the point I'm hoping is seen
> > in
> > > this.
> > >
> > >
> > >
> > > On Fri, Jul 12, 2019 at 10:08 AM Joe Witt <joe.witt@gmail.com> wrote:
> > >
> > > > to clarify user management for infra is not a prob.  it is an ldap
> > group.
> > > >
> > > > repo creation is self service as well amd group access is tied to that.
> > > >
> > > > release artifact is the source we produce.  this is typically
> > correlated
> > > to
> > > > a tag of the repo.  if we have all source in one repo it isnt clear to
> > me
> > > > how we can maintain that.
> > > >
> > > > in any event im not making a statement of whether to do many repos or
> > > not.
> > > > just correcting some potentially misleading claims.
> > > >
> > > > thanks
> > > >
> > > > On Fri, Jul 12, 2019, 12:01 PM Adam Taft <adam@adamtaft.com> wrote:
> > > >
> > > > > Just as a point of discussion, I'm not entirely sure that splitting
> > > into
> > > > > multiple physical git repositories is actually adding any value.
 I
> > > think
> > > > > it's worth consideration that all the (good) changes being proposed
> > are
> > > > > done under a single mono-repository model.
> > > > >
> > > > > If we split into multiple repositories, you have substantially
> > > increased
> > > > > the infra surface area. User account management overhead goes up.
> > > Support
> > > > > from the infra team goes up. JIRA issue management goes up,
> > > > > misfiled/miscategorized issues become common. It becomes harder for
> > > > > community members to interact and engage with the project, steeper
> > > > learning
> > > > > curve for new contributors. There are more "side channel"
> > conversations
> > > > and
> > > > > less transparency into the project as a whole. Git history is much
> > > harder
> > > > > (or impossible) to follow across the entire project. Tracking down
> > bugs
> > > > and
> > > > > performing git blame or git bisect becomes hard.
> > > > >
> > > > > There's nothing really stopping all of these changes from occurring
> > in
> > > > the
> > > > > existing repo, we don't have to have a maven pom.xml in the root
of
> > the
> > > > > project repository. It's much easier for contributors to just clone
a
> > > > > single repository, read the README at the root, and get oriented
to
> > the
> > > > > project layout.  Output artifacts can still be versioned differently
> > > (api
> > > > > can have a different version from extensions).  "Splitting out"
> > modules
> > > > can
> > > > > still happen in the mono-repository.  Jenkins and friends can be
> > taught
> > > > the
> > > > > project layout.
> > > > >
> > > > > tl;dr - The changes being proposed can be done in a single
> > repository.
> > > > > Splitting into multiple repositories is adding overhead on multiple
> > > > levels,
> > > > > which might be a sneaky form of muda. [1]
> > > > >
> > > > > Thanks for reading,
> > > > > Adam
> > > > >
> > > > > [1] https://dzone.com/articles/seven-wastes-software
> > > > >
> > > > >
> > > > > On Thu, Jul 11, 2019 at 11:01 AM Otto Fowler <
> > ottobackwards@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > I agree that this looks great. I think Mike’s idea is worth
> > > considering
> > > > > as
> > > > > > well. I would hope, that as part of this effort some thought
will
> > be
> > > > > given
> > > > > > to enhancing the developer documentation around the modules
would
> > be
> > > > > given
> > > > > > as well.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On July 10, 2019 at 18:15:21, Mike Thomsen (mikerthomsen@gmail.com
> > )
> > > > > wrote:
> > > > > >
> > > > > > I agree. It's very well thought out. One change to consider
is
> > > > splitting
> > > > > > the extensions further into two separate repos. One that would
> > serve
> > > > as a
> > > > > > standard library of sorts for other component developers and
> > another
> > > > that
> > > > > > would include everything else. Things like the Record API would
go
> > > into
> > > > > the
> > > > > > former so that we could have a more conservative release schedule
> > > going
> > > > > > forward with those components.
> > > > > >
> > > > > > On Wed, Jul 10, 2019 at 4:17 PM Andy LoPresto <
> > alopresto@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks Kevin, this looks really promising.
> > > > > > >
> > > > > > > Updating the link here as I think the page may have moved:
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/NIFI/NiFi+Project+and+Repository+Restructuring
> > > > > > > <
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/NIFI/NiFi+Project+and+Repository+Restructuring
> > > > > > > >
> > > > > > >
> > > > > > > Andy LoPresto
> > > > > > > alopresto@apache.org
> > > > > > > alopresto.apache@gmail.com
> > > > > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B
2F7D
> > EF69
> > > > > > >
> > > > > > > > On Jul 10, 2019, at 12:08 PM, Kevin Doran <kdoran@apache.org>
> > > > wrote:
> > > > > > > >
> > > > > > > > Hi NiFi Dev Community,
> > > > > > > >
> > > > > > > > Jeff Storck, Bryan Bende, and I have been collaborating
back
> > and
> > > > > forth
> > > > > > > > on a proposal for how to restructure the NiFi source
code into
> > > > > smaller
> > > > > > > > Maven projects and repositories based on the discussion
that
> > took
> > > > > > > > place awhile back on this thread. I'm reviving this
older
> > thread
> > > in
> > > > > > > > order to share that proposal with the community and
generate
> > > > farther
> > > > > > > > discussion about at solidifying a destination and
a plan for
> > how
> > > to
> > > > > > > > get there.
> > > > > > > >
> > > > > > > > Specifically, the proposal we've started working on
has three
> > > > parts:
> > > > > > > >
> > > > > > > > 1. Goals (more or less a summary of the earlier discussion
that
> > > > took
> > > > > > > > place on this thread)
> > > > > > > > 2. Proposed end state of the new Maven project and
repository
> > > > > structure
> > > > > > > > 3. Proposed approach for how to get from where we
are today to
> > > the
> > > > > > > > desired end state
> > > > > > > >
> > > > > > > > The proposal is on the Apache NiFi Wiki [1], so that
we can all
> > > > > > > > collaborate on it or leave comments there.
> > > > > > > >
> > > > > > > > [1]
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/NIFIREG/NiFi+Project+and+Repository+Restructuring
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Kevin, Jeff, and Bryan
> > > > > > > >
> > > > > > > > On Thu, May 30, 2019 at 1:31 PM Kevin Doran <kdoran@apache.org
> > >
> > > > > wrote:
> > > > > > > >>
> > > > > > > >> I am also in favor of splitting the nifi maven
project up into
> > > > > smaller
> > > > > > > >> projects with independent release cycles in order
to decouple
> > > > > > > >> development at well defined boundaries/interfaces
and also to
> > > > > > > >> facilitate code reuse.
> > > > > > > >>
> > > > > > > >> In anticipation of eventually working towards
a NiFi 2.0 that
> > > > > > > >> introduces bigger changes for developers and users,
I've
> > started
> > > > > work
> > > > > > > >> on a nifi-commons project in which I've extracted
out some of
> > > the
> > > > > code
> > > > > > > >> that originally got ported from NiFi -> NiFi
Registry, and now
> > > > > exists
> > > > > > > >> as similar code in both projects, into a standalone
modular
> > > > library.
> > > > > > > >> That premilinary work is here on my personal github
account
> > for
> > > > now
> > > > > > > >> [1].
> > > > > > > >>
> > > > > > > >> So far, it only contains some security code in
a submodule,
> > and
> > > > is a
> > > > > > > >> WIP (more work coming when I have time), but the
idea is
> > > > > nifi-commons
> > > > > > > >> could have several libraries/modules and would
be released
> > > > > > > >> periodically to use across nifi and registry.
If we are
> > talking
> > > > > about
> > > > > > > >> spliting the nifi project into framework and extensions,
then
> > > > > > > >> nifi-commons might be a good home for code that
needs to be
> > > shared
> > > > > > > >> across those two sub projects as well, such as
the nifi-api
> > bits
> > > > Joe
> > > > > > > >> mentioned.
> > > > > > > >>
> > > > > > > >> As part of this larger effort, I would be happy
to help get a
> > > > > > > >> nifi-commons repository started in Apache where
we can move
> > > shared
> > > > > > > >> code such as nifi-api to prepare for splitting
nifi-framework
> > > and
> > > > > > > >> nifi-extensions. It also occurs to me that if
nifi-framework
> > and
> > > > > > > >> nifi-extensions are being released independently,
> > nifi-assembly
> > > > > should
> > > > > > > >> probably just become a project that pulls in and
assembles the
> > > > > latest
> > > > > > > >> releases of framework and extensions.
> > > > > > > >>
> > > > > > > >> Overall, I think this would be beneficial for
most of the work
> > > > going
> > > > > > > >> on in Apache NiFi, which would not have to cut
across these
> > > > > different
> > > > > > > >> project and therefore would be easier to code,
test, build,
> > and
> > > > > > > >> release. However, the level of difficulty will
increase for
> > > > changes
> > > > > > > >> that will need to span multiple projects, though
those are
> > fewer
> > > > in
> > > > > > > >> number, so overall I think it would be a net win
for the dev
> > > > > > > >> community.
> > > > > > > >>
> > > > > > > >> [1] https://github.com/kevdoran/nifi-commons
> > > > > > > >>
> > > > > > > >> Thanks,
> > > > > > > >> Kevin
> > > > > > > >>
> > > > > > > >> On Thu, May 30, 2019 at 12:17 PM Andy LoPresto
<
> > > > > alopresto@apache.org>
> > > > > > > wrote:
> > > > > > > >>>
> > > > > > > >>> I am a strong +1 on the separation and reducing
the build
> > time.
> > > > > With
> > > > > > > that in mind, I think the process I brought up yesterday
[1] of
> > > > signing
> > > > > > our
> > > > > > > artifacts with GPG as part of the Maven build is paramount,
> > because
> > > > we
> > > > > > > would now be consuming core code across multiple
> > > > projects/repositories,
> > > > > > so
> > > > > > > there is even less guarantee the code is coming from “us”.
> > > > > > > >>>
> > > > > > > >>> [1]
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E
> > > > > > > <
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://lists.apache.org/thread.html/5974971939c539c34148d494f11e8bcf0640c440ce5e7a768ee9db01@%3Cdev.nifi.apache.org%3E
> > > > > > > >
> > > > > > > >>>
> > > > > > > >>> Andy LoPresto
> > > > > > > >>> alopresto@apache.org
> > > > > > > >>> alopresto.apache@gmail.com
> > > > > > > >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4
BACE 3C6E F65B 2F7D
> > > > EF69
> > > > > > > >>>
> > > > > > > >>>> On May 30, 2019, at 9:15 AM, Brandon DeVries
<brd@jhu.edu>
> > > > wrote:
> > > > > > > >>>>
> > > > > > > >>>> In regards to "We 'could' also split out
the 'nifi-api'...",
> > > > NiFi
> > > > > > 2.0
> > > > > > > would
> > > > > > > >>>> also be a good time to look at more clearly
defining the
> > > > > separation
> > > > > > > between
> > > > > > > >>>> the UI and the framework. Where nifi-api
is the contract
> > > between
> > > > > the
> > > > > > > >>>> extensions and the framework, the NiFi
Rest api is the
> > > contract
> > > > > > > between the
> > > > > > > >>>> UI and framework... These pieces could
potentially be built
> > /
> > > > > > > deployed /
> > > > > > > >>>> updated independently.
> > > > > > > >>>>
> > > > > > > >>>> On Thu, May 30, 2019 at 11:39 AM Jeff
<jtswork@gmail.com>
> > > > wrote:
> > > > > > > >>>>
> > > > > > > >>>>> In the same category of challenges
that Peter pointed out,
> > it
> > > > > might
> > > > > > > be
> > > > > > > >>>>> difficult for Travis to build the
"framework" and
> > > "extensions"
> > > > > > > projects if
> > > > > > > >>>>> there are changes in a PR that affect
both projects.
> > > > > > > >>>>>
> > > > > > > >>>>> Is there a good way in Travis to have
the workspace/maven
> > > repo
> > > > > > shared
> > > > > > > >>>>> between projects in a single build?
> > > > > > > >>>>>
> > > > > > > >>>>> It's probably always in the direction
of the extensions
> > > project
> > > > > > > needing
> > > > > > > >>>>> something new to be added to the framework
project rather
> > > than
> > > > > the
> > > > > > > other
> > > > > > > >>>>> way around, but it'll be tricky to
get that working right
> > in
> > > > > Travis
> > > > > > > if it's
> > > > > > > >>>>> not possible to set up the Travis
build to know it needs to
> > > > > deploy
> > > > > > > the
> > > > > > > >>>>> framework project artifacts into a
maven repo that the
> > > > extension
> > > > > > > project
> > > > > > > >>>>> will use.
> > > > > > > >>>>>
> > > > > > > >>>>> One way might be to make sure that
changes to the framework
> > > > > project
> > > > > > > must be
> > > > > > > >>>>> in master before the extensions project
can make use of
> > them,
> > > > but
> > > > > > > that
> > > > > > > >>>>> would require a "default master" build
for the framework
> > > > project
> > > > > > > which
> > > > > > > >>>>> builds master after each commit, and
deploys the build
> > > > artifacts
> > > > > to
> > > > > > a
> > > > > > > >>>>> persistent maven repo that the extension
project builds can
> > > > > access.
> > > > > > > It
> > > > > > > >>>>> also makes project-spanning change-sets
take longer to
> > review
> > > > and
> > > > > > > get fully
> > > > > > > >>>>> committed to master.
> > > > > > > >>>>>
> > > > > > > >>>>> On Thu, May 30, 2019 at 11:23 AM Peter
Wicks (pwicks) <
> > > > > > > pwicks@micron.com>
> > > > > > > >>>>> wrote:
> > > > > > > >>>>>
> > > > > > > >>>>>> One more "not awesome" would be
that core changes that
> > > affect
> > > > > > > extensions
> > > > > > > >>>>>> will be a little harder to test.
If I make a core change
> > > that
> > > > > > > changes the
> > > > > > > >>>>>> signature of an interface/etc...
I'll need to do some
> > extra
> > > > work
> > > > > > to
> > > > > > > make
> > > > > > > >>>>>> sure I don't break extensions
that use it.
> > > > > > > >>>>>>
> > > > > > > >>>>>> Still worth it, just one more
thing to mention.
> > > > > > > >>>>>>
> > > > > > > >>>>>> -----Original Message-----
> > > > > > > >>>>>> From: Joe Witt <joewitt@apache.org>
> > > > > > > >>>>>> Sent: Thursday, May 30, 2019 9:19
AM
> > > > > > > >>>>>> To: dev@nifi.apache.org
> > > > > > > >>>>>> Subject: [EXT] [discuss] Splitting
NiFi framework and
> > > > extension
> > > > > > > repos and
> > > > > > > >>>>>> releases
> > > > > > > >>>>>>
> > > > > > > >>>>>> Team,
> > > > > > > >>>>>>
> > > > > > > >>>>>> We've discussed this a bit over
the years in various forms
> > > but
> > > > > it
> > > > > > > again
> > > > > > > >>>>>> seems time to progress this topic
and enough has changed I
> > > > think
> > > > > > to
> > > > > > > >>>>> warrant
> > > > > > > >>>>>> it.
> > > > > > > >>>>>>
> > > > > > > >>>>>> Tensions:
> > > > > > > >>>>>> 1) Our build times take too long.
In travis-ci for
> > instance
> > > it
> > > > > > > takes 40
> > > > > > > >>>>>> minutes when it works.
> > > > > > > >>>>>> 2) The number of builds we do
has increased. We do
> > us/jp/fr
> > > > > builds
> > > > > > > on
> > > > > > > >>>>>> open and oracle JDKs. That is
6 builds.
> > > > > > > >>>>>> 3) We want to add Java 11 support
such that one could
> > build
> > > > > with 8
> > > > > > > or 11
> > > > > > > >>>>>> and the above still apply. The
becomes 6 builds.
> > > > > > > >>>>>> 4) With the progress in NiFi registry
we can now load
> > > > artifacts
> > > > > > > there and
> > > > > > > >>>>>> could pull them into NiFi. And
this integration will only
> > > get
> > > > > > > better.
> > > > > > > >>>>>> 5) The NiFi build is too huge
and cannot grow any longer
> > or
> > > > else
> > > > > > we
> > > > > > > >>>>> cannot
> > > > > > > >>>>>> upload convenience binaries.
> > > > > > > >>>>>>
> > > > > > > >>>>>> We cannot solve all the things
just yet but we can make
> > > > > progress.
> > > > > > I
> > > > > > > >>>>>> suggest we split apart the NiFi
'framework/application' in
> > > its
> > > > > own
> > > > > > > >>>>> release
> > > > > > > >>>>>> cycle and code repository from
the 'nifi extensions' into
> > > its
> > > > > own
> > > > > > > >>>>>> repository and release cycle.
The NiFi release would still
> > > > pull
> > > > > in
> > > > > > > a
> > > > > > > >>>>>> specific set of extension bundles
so to our end users at
> > > this
> > > > > time
> > > > > > > there
> > > > > > > >>>>> is
> > > > > > > >>>>>> no change. In the future we could
also just stop including
> > > the
> > > > > > > extensions
> > > > > > > >>>>>> in nifi the application and they
could be sourced at
> > runtime
> > > > as
> > > > > > > needed
> > > > > > > >>>>> from
> > > > > > > >>>>>> the registry (call that a NiFi
2.x thing).
> > > > > > > >>>>>>
> > > > > > > >>>>>> Why does this help?
> > > > > > > >>>>>> - Builds would only take as long
as just extensions take
> > or
> > > > just
> > > > > > > core/app
> > > > > > > >>>>>> takes. This reduces time for each
change cycle and reduces
> > > > load
> > > > > on
> > > > > > > >>>>>> travis-ci which runs the same
tests over and over and over
> > > for
> > > > > > each
> > > > > > > pull
> > > > > > > >>>>>> request/push regardless of whether
it was an extension or
> > > > core.
> > > > > > > >>>>>>
> > > > > > > >>>>>> - It moves us toward the direction
we're heading anyway
> > > > whereby
> > > > > > > >>>>> extensions
> > > > > > > >>>>>> can have their own lifecycle from
the framework/app
> > itself.
> > > > > > > >>>>>>
> > > > > > > >>>>>> How is this not awesome:
> > > > > > > >>>>>> - Doesn't yet solve for the large
builds problem. I think
> > > > we'll
> > > > > > get
> > > > > > > >>>>> there
> > > > > > > >>>>>> with a NiFi 2.x release which
fully leverages
> > nifi-registry
> > > > for
> > > > > > > retrieval
> > > > > > > >>>>>> of all extensions.
> > > > > > > >>>>>> - Adds another 'thing we need
to do a release cycle for'.
> > > This
> > > > > is
> > > > > > > >>>>>> generally unpleasant but it is
paid for once a release
> > cycle
> > > > and
> > > > > > it
> > > > > > > does
> > > > > > > >>>>>> allow us to release independently
for new cool
> > > > extensions/fixes
> > > > > > > apart
> > > > > > > >>>>> from
> > > > > > > >>>>>> the framework itself.
> > > > > > > >>>>>>
> > > > > > > >>>>>> Would be great to hear others
thoughts if they too feel it
> > > is
> > > > > time
> > > > > > > to
> > > > > > > >>>>> make
> > > > > > > >>>>>> this happen.
> > > > > > > >>>>>>
> > > > > > > >>>>>> Thanks
> > > > > > > >>>>>> Joe
> > > > > > > >>>>>>
> > > > > > > >>>>>
> > > > > > > >>>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >

Mime
View raw message