flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Piotr Nowojski <pi...@data-artisans.com>
Subject Re: java.lang.NoSuchMethodError and dependencies problem
Date Fri, 20 Oct 2017 11:46:06 GMT
But you said

> this seems to work as mvn dependency:tree -Ddetail=true only shows 1.14

To avoid this error that you describe I think that you have to ensure, that no 1.14 commons-compress
comes from your application, because it can conflict with 1.4.1 used by flink cluster.

By shading I meant that you could shade/relocate 1.14 usages in your application, so that
they don’t collide with Flink’s 1.4.1.

Piotrek

> On 19 Oct 2017, at 19:58, r. r. <robert@abv.bg> wrote:
> 
> Thanks, Piotr
> but my app code is self-contained in a fat-jar with maven-shade, so why would the class
path affect this?
> 
> by shade commons-compress do you mean :
> <!-- <exclude>org.apache.commons:commons-compress</exclude> -->
> it doesn't have effect either
> 
> as a last resort i may try to rebuild Flink to use 1.14, but don't want to go there yet
=/
> 
> 
> Best regards
> 
> 
> 
> 
> 
> 
>> -------- Оригинално писмо --------
> 
>> От: Piotr Nowojski piotr@data-artisans.com
> 
>> Относно: Re: java.lang.NoSuchMethodError and dependencies problem
> 
>> До: "r. r." <robert@abv.bg>
> 
>> Изпратено на: 19.10.2017 20:04
> 
> 
> 
> 
>> I’m not 100% sure, so treat my answer with a grain of salt.
> 
>> 
> 
>> I think when you start the cluster this way, dependencies (some? all?) are being
loaded to the class path before loading user’s application. At that point, it doesn’t
matter whether you have excluded commons-compress 1.4.1 in yours application pom.xml. I’m
not sure if this is solvable in some way, or not.
> 
>> 
> 
>> Maybe as a walk around, you could shade commons-compress usages in your pom.xml?
> 
>> 
> 
>> Piotr Nowojski
> 
>> 
> 
>>> On 19 Oct 2017, at 17:36, r. r. <robert@abv.bg> wrote:
> 
>>> 
> 
>>> flink is started with bin/start-local.sh
> 
>>> 
> 
>>> there is no classpath variable in the environment; flink/lib/flink-dist_2.11-1.3.2.jar
contains commons-compress, still it should be overridden by the dependencyManagement directive
> 
>>> 
> 
>>> here is the stacktrace:
> 
>>> 
> 
>>> The program finished with the following exception:
> 
>>> 
> 
>>> org.apache.flink.client.program.ProgramInvocationException: The program execution
failed: Job execution failed.
> 
>>>    at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:478)
> 
>>>    at org.apache.flink.client.program.StandaloneClusterClient.submitJob(StandaloneClusterClient.java:105)
> 
>>>    at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:442)
> 
>>>    at org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:73)
> 
>>>    at com.foot.semantic.flink.PipelineJob.main(PipelineJob.java:73)
> 
>>>    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 
>>>    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 
>>>    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 
>>>    at java.lang.reflect.Method.invoke(Method.java:497)
> 
>>>    at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528)
> 
>>>    at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:419)
> 
>>>    at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:381)
> 
>>>    at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:838)
> 
>>>    at org.apache.flink.client.CliFrontend.run(CliFrontend.java:259)
> 
>>>    at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1086)
> 
>>>    at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1133)
> 
>>>    at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1130)
> 
>>>    at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
> 
>>>    at java.security.AccessController.doPrivileged(Native Method)
> 
>>>    at javax.security.auth.Subject.doAs(Subject.java:422)
> 
>>>    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> 
>>>    at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
> 
>>>    at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1130)
> 
>>> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution
failed.
> 
>>>    at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply$mcV$sp(JobManager.scala:933)
> 
>>>    at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:876)
> 
>>>    at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:876)
> 
>>>    at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
> 
>>>    at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
> 
>>>    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
> 
>>>    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
> 
>>>    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> 
>>>    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> 
>>>    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 
>>>    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 
>>> Caused by: java.lang.NoSuchMethodError: org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String;
> 
>>>    at org.apache.tika.parser.pkg.ZipContainerDetector.detectArchiveFormat(ZipContainerDetector.java:114)
> 
>>>    at org.apache.tika.parser.pkg.ZipContainerDetector.detect(ZipContainerDetector.java:85)
> 
>>>    at org.apache.tika.detect.CompositeDetector.detect(CompositeDetector.java:77)
> 
>>>    at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:115)
> 
>>>    at com.foot.semantic.gafe.flink.services.TikaService.parse(TikaService.java:83)
> 
>>>    at com.foot.semantic.gafe.flink.services.TikaService.parse(TikaService.java:59)
> 
>>>    at com.foot.semantic.flink.uima.TikaReader.process(TikaReader.java:137)
> 
>>>    at com.foot.semantic.flink.uima.TikaReader.getNext(TikaReader.java:82)
> 
>>>    at org.apache.uima.fit.pipeline.SimplePipeline.runPipeline(SimplePipeline.java:175)
> 
>>>    at com.foot.semantic.flink.PipelineJob$1.map(PipelineJob.java:63)
> 
>>>    at com.foot.semantic.flink.PipelineJob$1.map(PipelineJob.java:37)
> 
>>>    at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41)
> 
>>>    at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:528)
> 
>>>    at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:503)
> 
>>>    at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:483)
> 
>>>    at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:891)
> 
>>>    at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:869)
> 
>>>    at org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collect(StreamSourceContexts.java:103)
> 
>>>    at org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collectWithTimestamp(StreamSourceContexts.java:110)
> 
>>>    at org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecordWithTimestamp(AbstractFetcher.java:269)
> 
>>>    at org.apache.flink.streaming.connectors.kafka.internal.Kafka010Fetcher.emitRecord(Kafka010Fetcher.java:86)
> 
>>>    at org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.runFetchLoop(Kafka09Fetcher.java:152)
> 
>>>    at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:483)
> 
>>>    at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
> 
>>>    at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:55)
> 
>>>    at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:95)
> 
>>>    at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:263)
> 
>>>    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
> 
>>>    at java.lang.Thread.run(Thread.java:745)
> 
>>> 
> 
>>> 
> 
>>> 
> 
>>> 
> 
>>> 
> 
>>> 
> 
>>> 
> 
>>> 
> 
>>> 
> 
>>>> -------- Оригинално писмо --------
> 
>>> 
> 
>>>> От: Piotr Nowojski piotr@data-artisans.com
> 
>>> 
> 
>>>> Относно: Re: java.lang.NoSuchMethodError and dependencies problem
> 
>>> 
> 
>>>> До: "r. r." <robert@abv.bg>
> 
>>> 
> 
>>>> Изпратено на: 19.10.2017 18:00
> 
>>> 
> 
>>> 
> 
>>> 
> 
>>> 
> 
>>>> Hi,
> 
>>> 
> 
>>>> 
> 
>>> 
> 
>>>> What is the full stack trace of the error?
> 
>>> 
> 
>>>> Are you sure that there is no commons-compresss somewhere in the classpath
(like in the lib directory)? How are you running your Flink cluster?
> 
>>> 
> 
>>>> 
> 
>>> 
> 
>>>> Piotrek
> 
>>> 
> 
>>>> 
> 
>>> 
> 
>>>>> On 19 Oct 2017, at 13:34, r. r. <robert@abv.bg> wrote:
> 
>>> 
> 
>>>>> 
> 
>>> 
> 
>>>>> Hello
> 
>>> 
> 
>>>>> I have a job that runs an Apache Tika pipeline and it fails with "Caused
by: java.lang.NoSuchMethodError: org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String;"
> 
>>> 
> 
>>>>> 
> 
>>> 
> 
>>>>> Flink includes commons-compress 1.4.1, while Tika needs 1.14. 
> 
>>> 
> 
>>>>> I also have Apache Avro in the project with commons-compress at 1.8.1,
so I force 1.14 with 
> 
>>> 
> 
>>>>> 
> 
>>> 
> 
>>>>>   <dependencyManagement>
> 
>>> 
> 
>>>>>       <dependencies>
> 
>>> 
> 
>>>>>           <dependency>
> 
>>> 
> 
>>>>>               <groupId>org.apache.commons</groupId>
> 
>>> 
> 
>>>>>               <artifactId>commons-compress</artifactId>
> 
>>> 
> 
>>>>>               <version>1.14</version>
> 
>>> 
> 
>>>>>           </dependency>
> 
>>> 
> 
>>>>>       </dependencies>
> 
>>> 
> 
>>>>>   </dependencyManagement>
> 
>>> 
> 
>>>>> 
> 
>>> 
> 
>>>>> this seems to work as mvn dependency:tree -Ddetail=true only shows 1.14
and after purge, the local maven repo also only contains 1.14
> 
>>> 
> 
>>>>> 
> 
>>> 
> 
>>>>> yet, after i deploy the job and it reads an Avro package from kafka and
passes it to Tika, it fails with the error above, which leads me to think it somehow uses
commons-compress at a version prior to 1.14, because method 'detect' is not present in older
versions
> 
>>> 
> 
>>>>> 
> 
>>> 
> 
>>>>> I excluded/included it from the fat-jar
> 
>>> 
> 
>>>>> <exclude>org.apache.commons:commons-compress</exclude>
> 
>>> 
> 
>>>>> still the same problem
> 
>>> 
> 
>>>>> 
> 
>>> 
> 
>>>>> thanks for any hints!
> 
>>> 
> 
>>>>> 
> 
>>> 
> 
>>>>> 


Mime
View raw message