nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <joe.w...@gmail.com>
Subject Re: [DISCUSS] Provided dependency on bcprov-jdk15on
Date Tue, 01 Nov 2016 16:19:49 GMT
Mike

Yep - that will need to get addressed/improved as well.  First step is
get the root classpath cleaned up again (NIFI-2971) and the next step
is to understand and address the excessive declaration of dependencies
which leads to duplication on such a scale.

Thanks
Joe

On Tue, Nov 1, 2016 at 12:15 PM, Michael Moser <moser.mw@gmail.com> wrote:
> I would like to point out that some very common dependencies have been
> placed into the nifi-standard-services-api-nar.  This nar is a
> Nar-Dependency-Id of several other nars.  Many of these other nars also
> include the same dependencies, which would not be used by the nar class
> loader.
>
> In particular, nifi-standard-services-api-nar contains nifi-utils,
> nifi-processor-utils, and nifi-security-utils.  It also contains the
> bcprov-jdk15on and bcpkix-jdk15on jars.
>
> Then looking at nifi-standard-nar, nifi-aws-nar, nifi-kafka-0-10-nar and
> more, they also contain these same jars in their bundled-dependencies.
> Their Nar-Dependency-Id is nifi-standard-services-api-nar.
>
> -- Mike
>
>
> On Tue, Nov 1, 2016 at 1:50 PM, Aldrin Piri <aldrinpiri@gmail.com> wrote:
>
>> I think in light of the fact that the bcprov item was an accidental
>> inclusion and not explicitly meant to be handled as such in the framework
>> reorganizing seems like the best option as well.  The original intent of
>> the PR was to minimize that footprint where we were double dipping given
>> the inclusion of nifi-processor-utils.  However, if we manage to shuffle
>> items around a bit, the net size reduction could ultimately be the same.
>> Most importantly, from a maintenance perspective, bundles will only include
>> the library in the NARs that need it which allows that dependency graph to
>> be significantly more comprehensible than the scope tweaking originally
>> presented.
>>
>> Will this also work to adjust the dependencies of nifi-processor-utils (and
>> potentially nifi-security-utils)?  Currently, the archetype also provides
>> nifi-processor-utils, which could lead to similar conflicts for folks that
>> were looking to make use of different versions of JARs within a given NAR.
>>
>> I think the listing Joe provided in the ticket is great and we should be
>> mindful of any deltas while performing each release.  For the reasons
>> highlighted in this chain and on that issue, one jar inclusion could be
>> quite detrimental and cause some difficult issues to those that have
>> created extensions.
>>
>> On Mon, Oct 31, 2016 at 9:39 PM, Joe Witt <joe.witt@gmail.com> wrote:
>>
>> > So...I might be a bit of a flip flopper...
>> >
>> > Had some off-list discussions with a couple people and they really
>> > disliked this idea.  They, rightfully, pointed out how this
>> > establishes a problematic precedent, how it is really only being done
>> > to save space, and that while perhaps not able to pair down to 200MB
>> > we can likely do more analysis of the dependency graph to improve the
>> > situation.  The real issue that kicked this off was the finding that
>> > bcprov (and commons-lang3) had made their way into the root of our
>> > classloader (which impacts all classloaders) and this was introduced
>> > with the important NIFI-1831 work.  So I've created [1] to address
>> > that and return the application classloader to the minimal set of jars
>> > necessary which should basically be the runtime application launcher,
>> > the logging libraries, and the nifi-api.
>> >
>> > The ultimate solution to addressing the distribution size concerns
>> > will be to address it via the extension registry.
>> >
>> > [1] https://issues.apache.org/jira/browse/NIFI-2971
>> >
>> > Thanks
>> > Joe
>> >
>> > On Mon, Oct 31, 2016 at 8:51 PM, Andre <andre-lists@fucs.org> wrote:
>> > > +1 as well.
>> > >
>> > > On Tue, Nov 1, 2016 at 2:41 AM, Joe Witt <joe.witt@gmail.com> wrote:
>> > >
>> > >> Team,
>> > >>
>> > >> In the 1.x line bcprov made its way into the root of the classpath/lib
>> > >> folder to support the encrypted sensitive properties features.  This
>> > >> can be reworked to isolate it a bit more if we need to.
>> > >>
>> > >> However, in [1] it has been observed that we have bcprov dependencies
>> > >> in about 53 places with each instance taking about 4MB of space for
a
>> > >> total hit of about 200MB of bcprov.
>> > >>
>> > >> [1] proposes to make bcprov-jdk15on and bcpkix-jdk15on part of the
>> > >> core provided list of things right along side the
>> > >> logback/slf4j/logging interfaces. This means all things will have
>> > >> access to these going forward and it means it impacts our version
>> > >> compatibility.  Given that it would be a standard/provided thing if
we
>> > >> want to upgrade versions or swap it out with something else we could
>> > >> break extensions that then chose to depend on it.  We are not in
>> > >> control of the bcprov api just like we're not in control of the
>> > >> logging APIs.  If there was some important security related fix we
>> > >> needed from bcprov but changing also pulled in api changes for them
it
>> > >> could break our extensions.
>> > >>
>> > >> Even with all this said, given the nature, importance, and size
>> > >> benefit, I am in favor of NIFI-2954.  But, would like to highlight
>> > >> this in case others have perspective they'd like to share.
>> > >>
>> > >> [1] https://issues.apache.org/jira/browse/NIFI-2954
>> > >>
>> > >> Thanks
>> > >> Joe
>> > >>
>> >
>>

Mime
View raw message