nifi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joseph Witt (JIRA)" <>
Subject [jira] [Commented] (NIFI-189) Collapse commons without dependency into a single library
Date Sat, 20 Dec 2014 05:47:14 GMT


Joseph Witt commented on NIFI-189:

Considering the following comments from the dev list mailing thread as the review in this

I'm not sure if this is the most appropriate forum or I should have
just written a Jira ticket, but here goes.

I believe we should consolidate the number of artifacts we have in the
nifi/commons module.  We create three jars that contain just 1 class
each and there are three more jars with 3 or fewer classes in them.
This makes it annoying (especially for beginners) to find the location
of classes that you need and slightly bloats our footprint for number
of artifacts that nifi create.  I believe we can improve this.

I analyzed all of the nar-bundles to find where each common library
was used.  Several are used by many framework, services, and
processors bundles already, so consolidating these common jars is a
no-brainer.  Other jars that are used more sparingly contain just 1 or
2 classes, so it really will have minimal impact to consolidate them
even if the classes aren't needed by a nar.

So, I propose we consolidate these artifacts into the nifi-utils
artifact.  The number in (parentheses) is the number of classes in

core-flowfile-attributes (2)
flowfile-packager (9)
naive-search-ring-buffer (1)
nifi-file-utils (1)
nifi-logging-utils (1)
nifi-properties (2)
nifi-security-utils (5)
nifi-socket-utils (24)
nifi-stream-utils (17)
processor-utilities (3) (this would also resolve why the name doesn't
start with "nifi")

nifi-utils would go from 24 classes to 89 classes.

nifi-web-utils (3), remote-communications-utils (13), and search-utils
(5) I did not include because their use is limited to just one or two

-- Mike

Do we know how many transitive dependencies this ends up mixing together?

I bring it up because that's often a reason for splitting a small
number of classes into their own module. For example, if I care about
socket-based data flow maybe I don't need the dependancies utilities
related to file-based data flow. I'll try to take a look at the actual
modules, but I thought I would throw that out there for others to
think on.

One thing I've seen work well is creating a dependency aggregator
module for users that don't care about the extra dependencies.


Joey makes a good point.  Without nifi-socket-utils, the transitive
dependencies will be commons-codec, commons-compress and
commons-lang3.  slf4j-api is also in there but that's marked provided
by all of NiFi.

When you add nifi-socket-utils to the equation, that adds commons-io
and commons-net.  Interestingly, nifi-socket-utils also depends on
three of the other nifi-*-utils.  So if you need nifi-socket-utils,
you also get nifi-properties, nifi-logging-utils, and nifi-utils

It's probably worth leaving nifi-socket-utils separate for now.  It
had the biggest footprint of all of the utils to begin with.

flowfile-packager is the only one that pulls in commons-compress.
nifi-file-utils is the only one that pulls in commons-codec (though
that dependency could be removed with a clever refactor of
computeMd5Digest(File file) using just the JDK).

-- Mike
You didn't fall too far, Joe.  core-flowfile-attributes,
naive-search-ring-buffer, nifi-properties, nifi-stream-utils, and
processor-utilities have no dependencies.  And nifi-logging-utils only
depends on the provided slf4j-api.  Just sayin'  ;)

-- Mike

Those sound like really good candidates for consolidation. It might
also be worth looking at the dependency graph to find a lot of
co-occurence. If every module that depends on A also depends on B,
there's less of an argument to keep them separate.


These probably can be cleaned up some now, as we now have a much more unified build, with
a high level dependency management.
Careful, though: nifi-properties cannot be merged with anything else! It is on the root of
the lib/ directory, so it cannot have any transitive dependencies at all.

> Collapse commons without dependency into a single library
> ---------------------------------------------------------
>                 Key: NIFI-189
>                 URL:
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Tools and Build
>            Reporter: Joseph Witt
>            Assignee: Joseph Witt
>            Priority: Minor
>              Labels: refactor
>             Fix For: 0.0.1
> Several folks commented on a thread requesting that we collapse the number of commons/libraries
that exist.  There are several which have no dependencies at all and have relatively few classes.
 Going to collapse these into a single library which is intended to never have dependencies
so it keeps its surface area low.  The packages for the classes will also be updated to more
accurately reflect what it is.

This message was sent by Atlassian JIRA

View raw message