flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Piotr Nowojski <pi...@data-artisans.com>
Subject Re: java.lang.NoSuchMethodError and dependencies problem
Date Thu, 19 Oct 2017 17:04:50 GMT
I’m not 100% sure, so treat my answer with a grain of salt.

I think when you start the cluster this way, dependencies (some? all?) are being loaded to
the class path before loading user’s application. At that point, it doesn’t matter whether
you have excluded commons-compress 1.4.1 in yours application pom.xml. I’m not sure if this
is solvable in some way, or not.

Maybe as a walk around, you could shade commons-compress usages in your pom.xml?

Piotr Nowojski

> On 19 Oct 2017, at 17:36, r. r. <robert@abv.bg> wrote:
> 
> flink is started with bin/start-local.sh
> 
> there is no classpath variable in the environment; flink/lib/flink-dist_2.11-1.3.2.jar
contains commons-compress, still it should be overridden by the dependencyManagement directive
> 
> here is the stacktrace:
> 
> The program finished with the following exception:
> 
> org.apache.flink.client.program.ProgramInvocationException: The program execution failed:
Job execution failed.
>     at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:478)
>     at org.apache.flink.client.program.StandaloneClusterClient.submitJob(StandaloneClusterClient.java:105)
>     at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:442)
>     at org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:73)
>     at com.foot.semantic.flink.PipelineJob.main(PipelineJob.java:73)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:497)
>     at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528)
>     at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:419)
>     at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:381)
>     at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:838)
>     at org.apache.flink.client.CliFrontend.run(CliFrontend.java:259)
>     at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1086)
>     at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1133)
>     at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1130)
>     at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:422)
>     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>     at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
>     at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1130)
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
>     at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply$mcV$sp(JobManager.scala:933)
>     at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:876)
>     at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:876)
>     at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
>     at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
>     at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
>     at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
>     at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>     at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>     at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>     at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: java.lang.NoSuchMethodError: org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String;
>     at org.apache.tika.parser.pkg.ZipContainerDetector.detectArchiveFormat(ZipContainerDetector.java:114)
>     at org.apache.tika.parser.pkg.ZipContainerDetector.detect(ZipContainerDetector.java:85)
>     at org.apache.tika.detect.CompositeDetector.detect(CompositeDetector.java:77)
>     at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:115)
>     at com.foot.semantic.gafe.flink.services.TikaService.parse(TikaService.java:83)
>     at com.foot.semantic.gafe.flink.services.TikaService.parse(TikaService.java:59)
>     at com.foot.semantic.flink.uima.TikaReader.process(TikaReader.java:137)
>     at com.foot.semantic.flink.uima.TikaReader.getNext(TikaReader.java:82)
>     at org.apache.uima.fit.pipeline.SimplePipeline.runPipeline(SimplePipeline.java:175)
>     at com.foot.semantic.flink.PipelineJob$1.map(PipelineJob.java:63)
>     at com.foot.semantic.flink.PipelineJob$1.map(PipelineJob.java:37)
>     at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41)
>     at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:528)
>     at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:503)
>     at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:483)
>     at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:891)
>     at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:869)
>     at org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collect(StreamSourceContexts.java:103)
>     at org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collectWithTimestamp(StreamSourceContexts.java:110)
>     at org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecordWithTimestamp(AbstractFetcher.java:269)
>     at org.apache.flink.streaming.connectors.kafka.internal.Kafka010Fetcher.emitRecord(Kafka010Fetcher.java:86)
>     at org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.runFetchLoop(Kafka09Fetcher.java:152)
>     at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:483)
>     at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
>     at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:55)
>     at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:95)
>     at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:263)
>     at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
>     at java.lang.Thread.run(Thread.java:745)
> 
> 
> 
> 
> 
> 
> 
> 
> 
>> -------- Оригинално писмо --------
> 
>> От: Piotr Nowojski piotr@data-artisans.com
> 
>> Относно: Re: java.lang.NoSuchMethodError and dependencies problem
> 
>> До: "r. r." <robert@abv.bg>
> 
>> Изпратено на: 19.10.2017 18:00
> 
> 
> 
> 
>> Hi,
> 
>> 
> 
>> What is the full stack trace of the error?
> 
>> Are you sure that there is no commons-compresss somewhere in the classpath (like
in the lib directory)? How are you running your Flink cluster?
> 
>> 
> 
>> Piotrek
> 
>> 
> 
>>> On 19 Oct 2017, at 13:34, r. r. <robert@abv.bg> wrote:
> 
>>> 
> 
>>> Hello
> 
>>> I have a job that runs an Apache Tika pipeline and it fails with "Caused by:
java.lang.NoSuchMethodError: org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String;"
> 
>>> 
> 
>>> Flink includes commons-compress 1.4.1, while Tika needs 1.14. 
> 
>>> I also have Apache Avro in the project with commons-compress at 1.8.1, so I force
1.14 with 
> 
>>> 
> 
>>>    <dependencyManagement>
> 
>>>        <dependencies>
> 
>>>            <dependency>
> 
>>>                <groupId>org.apache.commons</groupId>
> 
>>>                <artifactId>commons-compress</artifactId>
> 
>>>                <version>1.14</version>
> 
>>>            </dependency>
> 
>>>        </dependencies>
> 
>>>    </dependencyManagement>
> 
>>> 
> 
>>> this seems to work as mvn dependency:tree -Ddetail=true only shows 1.14 and after
purge, the local maven repo also only contains 1.14
> 
>>> 
> 
>>> yet, after i deploy the job and it reads an Avro package from kafka and passes
it to Tika, it fails with the error above, which leads me to think it somehow uses commons-compress
at a version prior to 1.14, because method 'detect' is not present in older versions
> 
>>> 
> 
>>> I excluded/included it from the fat-jar
> 
>>> <exclude>org.apache.commons:commons-compress</exclude>
> 
>>> still the same problem
> 
>>> 
> 
>>> thanks for any hints!
> 
>>> 
> 
>>> 


Mime
View raw message