nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Russell Bateman <r...@windofkeltia.com>
Subject Re: Pulling out hair over slf4j...
Date Tue, 09 May 2017 20:17:01 GMT
My specific problem is that the Tika dependency is in an ancient code 
base (including /medical-filter/ which also figures in this example) 
built using ant. So, it's hard-linked and not using Maven. I'm thinking 
about how I can deal with this problem. I don't remember there being a 
notion of exclusion in ant, but there may have been. We have a huge 
allegiance to this old base atop which some 20-30 custom NiFi processors 
whose names begin with "Legacy" are built. These enabled our conversion 
from our old ETL framework to our new one that's NiFi-based. A handful 
of these legacy processors are still very heavily relied upon for the 
specialized things they do.

I'm pretty sure we completely understand this problem now. I'll just 
have to see how easily I can fix it starting with trying to use 
exclusion from /medical-filter/ which we do now bring in via Maven.

I thank all of you, Andrew in particular, for helping me nail this down 
and I hope this thread might serve others. (Actually, I rather hope no 
one else even has this trouble.)

Very best regards,

Russ

On 05/09/2017 11:42 AM, Andrew Psaltis wrote:
> Russell,
> I was going to echo the same thing Jeff mentioned. I built a little test
> app that uses the mave-share plugin and also exclude slf4j in the same way
> suggested earlier. This worked as expected, properly excluding the slf4j
> jar from the tika jar when shading my test app.
>
> On Tue, May 9, 2017 at 1:20 PM, Jeff <jtswork@gmail.com> wrote:
>
>> Tika, in the 1.0 tag, has slf4j-log4j12 [1] as a provided dependency,
>> version 1.5.6 (which I see gets brought in to your maven repo).  Later
>> versions of Tika are using maven-shade-plugin, but it doesn't look like 1.0
>> does.
>>
>> Have you tried excluding that dependency from tika-app?
>>
>> Oddly, tika-bundle uses a different version of slf4j-simple [2] for tests
>> (1.6.1).
>>
>> [1] https://github.com/apache/tika/blob/1.0/tika-app/pom.xml#L53
>> [2] https://github.com/apache/tika/blob/1.0/tika-bundle/pom.xml#L99
>>
>> On Tue, May 9, 2017 at 10:03 AM Russell Bateman <russ@windofkeltia.com>
>> wrote:
>>
>>> Thanks, Andrew.I haven't tried that before (a new diagnostic tool in my
>>> quiver).
>>>
>>> I did it, scraped the IntelliJ IDEA test console, then split lines on
>>> colons and grep'd for /slf4j/ to get this:
>>>
>>> *master ~/notes $ fgrep slf4j debug-console.log *
>>> :/home/russ/.m2/repository/org/slf4j/slf4j-api/1.7.25/
>> slf4j-api-1.7.25.jar
>>> :/home/russ/.m2/repository/org/slf4j/slf4j-simple/1.7.25/
>> slf4j-simple-1.7.25.jar
>>> :/home/russ/.m2/repository/org/slf4j/jcl-over-slf4j/1.7.
>> 25/jcl-over-slf4j-1.7.25.jar
>>> :/home/russ/.m2/repository/org/slf4j/log4j-over-slf4j/1.
>> 7.25/log4j-over-slf4j-1.7.25.jar
>>> :/home/russ/.m2/repository/org/slf4j/jul-to-slf4j/1.7.25/
>> jul-to-slf4j-1.7.25.jar
>>> [Loaded org.slf4j.LoggerFactory from
>>>
>>> file:/home/russ/.m2/repository/org/apache/tika/
>> tika-app/1.0/tika-app-1.0.jar]
>>> [Loaded org.slf4j.ILoggerFactory from
>>>
>>> file:/home/russ/.m2/repository/org/apache/tika/
>> tika-app/1.0/tika-app-1.0.jar]
>>> [Loaded org.slf4j.helpers.SubstituteLoggerFactory from
>>>
>>> file:/home/russ/.m2/repository/org/apache/tika/
>> tika-app/1.0/tika-app-1.0.jar]
>>> [Loaded org.slf4j.Logger from
>>>
>>> file:/home/russ/.m2/repository/org/apache/tika/
>> tika-app/1.0/tika-app-1.0.jar]
>>> [Loaded org.slf4j.spi.LoggerFactoryBinder from
>>>
>>> file:/home/russ/.m2/repository/org/apache/tika/
>> tika-app/1.0/tika-app-1.0.jar]
>>> [Loaded org.slf4j.impl.StaticLoggerBinder from
>>>
>>> file:/home/russ/.m2/repository/com/perfectsearchcorp/medical-
>> filter/195/medical-filter-195.jar]
>>> [Loaded org.slf4j.helpers.MessageFormatter from
>>>
>>> file:/home/russ/.m2/repository/org/apache/tika/
>> tika-app/1.0/tika-app-1.0.jar]
>>> java.lang.AssertionError: java.lang.NoSuchMethodError:
>>>
>>> org.slf4j.helpers.MessageFormatter.arrayFormat(
>> Ljava/lang/String;[Ljava/lang/Object;)Lorg/slf4j/helpers/FormattingTuple;
>>> Caused by: java.lang.NoSuchMethodError:
>>>
>>> org.slf4j.helpers.MessageFormatter.arrayFormat(
>> Ljava/lang/String;[Ljava/lang/Object;)Lorg/slf4j/helpers/FormattingTuple;
>>>
>>> I'm very concerned by Tika having hard-linked /slf4j/. Is that what's
>>> going on? I know from experience that I cannot use any other version of
>>> Tika.* Anyway, what does this tell you?
>>>
>>> I greatly appreciate your help and suggestions in this.
>>>
>>> Russ
>>>
>>> * I've been working on excerpting those bits of our code base that
>>> consume Tika to put them into a different project (and NAR). It's not
>>> been easy and I haven't finished yet doing it in the midst of pretty
>>> tight deadlines. My former colleague left things pretty interdependent
>>> in February when he left (and told me it was going to be a lot of work
>>> to separate what we call "legacy" processors out from the rest).
>>>
>>> On 05/08/2017 08:37 PM, Andrew Psaltis wrote:
>>>> Russell,
>>>> Sorry this is so much of a pain. I wonder if you can pass using the JVM
>>>> argument -verbose:class and then running the test can shed more light
>> on
>>>> which JAR is pulling it in.
>>>>
>>>> On Mon, May 8, 2017 at 5:14 PM, Russell Bateman <russ@windofkeltia.com
>>>> wrote:
>>>>
>>>>> I understand that no NiFi JAR comes with /slf4j/, but expects the
>>>>> consuming build to provide it. That's the assumption I've been going
>> on
>>> and
>>>>> it's why in my top-level /pom.xml/, I've specified [1.7.25] so that no
>>>>> quibbling over version can occur and I specify these JARs in the root
>>>>> /pom.xml/ so that no "renegade" submodule calls the dependency for its
>>> own.
>>>>> The problem is that some version of slf4j-api-x.y.z.jar doesn't
>> provide
>>> a
>>>>> MessageFormatter class containing an arrayFormat() method, right?
>>> Somehow,
>>>>> though, this impoverished version is the one bound to my code despite
>> my
>>>>> marking every dependency with Andrew's exclusion statements?
>>>>>
>>>>> There is nothing giving this away; in IntelliJ IDEA, under External
>>>>> Libraries in the Project pane, I see only 1.7.25 of these libraries,
>> no
>>>>> additional versions. I wonder if I should see more versions if in fact
>>>>> that's what's going on?
>>>>>
>>>>> I tried wiping /~/.m2/repository/org/slf4j/, after mvn clean compile
>> of
>>> my
>>>>> project, I see all those versions have come back to
>>>>> /~/.m2/repository/org/slf4j/. So, at the root of my project, I created
>>>>> /hard-repository/, and stuffed it with the slf4j-relevant JARs
>> arranged
>>> in
>>>>> their repository accoutrements (paths, files, everything), then added
>>>>>
>>>>>      <repositories>
>>>>>         <repository>
>>>>>           <id>slf4j</id>
>>>>>           <name>slf4j repo</name>
>>>>>
>>>   <url>file://${project.build.directory}/hard-repository/maven_repo</url>
>>>>>        </repository>
>>>>>      </repositories>
>>>>>
>>>>> ...first, so that this dependency would be satisfied--not that this
>>> should
>>>>> be necessary. So, I took this back out as superfluous.
>>>>>
>>>>> To me, this means that something else I'm linking has already
>>> hard-linked
>>>>> some version of slf4j-api older than (maybe 1.7.x--whenever
>>>>> MessageFormatter acquired this new method) which would prevent me from
>>>>> getting 1.7.25 and yet, IntelliJ's External Libraries list and,
>>> examining
>>>>> the module dependencies tab for each individual module I see only
>>> 1.7.25.
>>>>> I am stumped. I guess I could attempt to link in the NiFi test
>> framework
>>>>> in such a way as to be able to debug down into it better to watch
>> fail.
>>>>> I've stepped down in using IDEA's decompiler, which I would think
>>> adequate,
>>>>> but I don't see exactly what's amiss.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 05/08/2017 11:10 AM, Bryan Bende wrote:
>>>>>
>>>>>> Ok thanks for all the info, I am not totally sure what is going,
but
>> I
>>>>>> can tell you how the logging JARs are setup in NiFi and maybe that
>>>>>> will shed some light on things...
>>>>>>
>>>>>> In the root pom for NiFi there is a dependencyManagement section
that
>>>>>> declares slf4j-api, jul-to-slf4j, log4j-over-slf4j, and
>> jcl-over-slf4j
>>>>>> all as provided dependencies to ensure no NARs actually bundle these
>>>>>> since they will be provided directly in the lib directory and
>>>>>> available to all NARs, this also forces them all to
>>>>>> ${org.slf4j.version} which in master is 1.7.25:
>>>>>>
>>>>>> <dependency>
>>>>>>        <groupId>org.slf4j</groupId>
>>>>>>        <artifactId>jcl-over-slf4j</artifactId>
>>>>>>        <version>${org.slf4j.version}</version>
>>>>>>        <scope>provided</scope>
>>>>>> </dependency>
>>>>>> <dependency>
>>>>>>        <groupId>org.slf4j</groupId>
>>>>>>        <artifactId>log4j-over-slf4j</artifactId>
>>>>>>        <version>${org.slf4j.version}</version>
>>>>>>        <scope>provided</scope>
>>>>>> </dependency>
>>>>>> <dependency>
>>>>>>        <groupId>org.slf4j</groupId>
>>>>>>        <artifactId>jul-to-slf4j</artifactId>
>>>>>>        <version>${org.slf4j.version}</version>
>>>>>>        <scope>provided</scope>
>>>>>> </dependency>
>>>>>> <dependency>
>>>>>>        <groupId>org.slf4j</groupId>
>>>>>>        <artifactId>slf4j-api</artifactId>
>>>>>>        <version>${org.slf4j.version}</version>
>>>>>>        <scope>provided</scope>
>>>>>> </dependency>
>>>>>>
>>>>>>
>>>>>> Then the nifi-assembly pom.xml declares these using compile scope
and
>>>>>> puts them in the lib directory:
>>>>>>
>>>>>> <dependency>
>>>>>>        <groupId>org.slf4j</groupId>
>>>>>>        <artifactId>jcl-over-slf4j</artifactId>
>>>>>>        <scope>compile</scope>
>>>>>> </dependency>
>>>>>> <dependency>
>>>>>>        <groupId>org.slf4j</groupId>
>>>>>>        <artifactId>jul-to-slf4j</artifactId>
>>>>>>        <scope>compile</scope>
>>>>>> </dependency>
>>>>>> <dependency>
>>>>>>        <groupId>org.slf4j</groupId>
>>>>>>        <artifactId>log4j-over-slf4j</artifactId>
>>>>>>        <scope>compile</scope>
>>>>>> </dependency>
>>>>>> <dependency>
>>>>>>        <groupId>org.slf4j</groupId>
>>>>>>        <artifactId>slf4j-api</artifactId>
>>>>>>        <scope>compile</scope>
>>>>>> </dependency>
>>>>>>
>>>>>> Now for Unit tests, the root pom has a regular dependencies section
>>>>>> that has slf4j-simple declared with test scope:
>>>>>>
>>>>>>        <dependency>
>>>>>>            <groupId>org.slf4j</groupId>
>>>>>>            <artifactId>slf4j-simple</artifactId>
>>>>>>            <scope>test</scope>
>>>>>>        </dependency>
>>>>>>
>>>>>> So every NAR that is with in the NiFi project will have slf4j-simples
>>>>>> available for unit tests.
>>>>>>
>>>>>> Andrew Psaltis also mentions a good point about the different
>>>>>> versions, I believe we just upgraded to 1.7.25 for the 1.2.0 release,
>>>>>> but previous were on 1.7.12, but you should double check that.
>>>>>>
>>>>>>
>>>>>>
>>>
>
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message