flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aris kol <gizera...@hotmail.com>
Subject Re: NoClassDefFoundError with ElasticsearchSink on Yarn
Date Thu, 01 Sep 2016 10:22:43 GMT
Classic problem of every uber-jar containing Hadoop dependencies and being deployed on Yarn.

What actually happens is that some Hadoop dependency relies on an old version of guava (11
in this case), which doesn't have the method. You may have assembled your fat-jar properly,
but because Hadoop deps get introduced to your classpath before your own, you invoke the method
using the guava 11 version of the class.

I fixed that by adding this line:


++ Seq(assemblyShadeRules in assembly := Seq(ShadeRule.rename("com.google.common.**" ->
"shaded.@1").inAll))

on the artefact that gets deployed on flink.

What this practically does is to shade the guava dependencies. Instead of containing references
to com.google.common your build will reference shaded.com.google.common and as a result it
will use the proper class in your fat jar.
Get a bit creative with the name (ie use shadedhausmann instead of shaded) to avoid colliding
with external deps shading stuff (something you have to do when using guava, joda, jackson
etc).

Let me know if this helped.

Aris

________________________________
From: Steffen Hausmann <steffen@hausmann-family.de>
Sent: Thursday, September 1, 2016 8:58 AM
To: user@flink.apache.org
Subject: NoClassDefFoundError with ElasticsearchSink on Yarn

Hi there,

I'm running a flink program that reads from a Kinesis stream and
eventually writes to an Elasticsearch2 sink. When I'm running the
program locally from the IDE, everything seems to work fine, but when
I'm executing the same program on an EMR cluster with Yarn, a
NoClassDefFoundError occurs:java.lang.NoSuchMethodError:

com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
        at
org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.java:190)
        at
org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:133)
        at
org.apache.flink.streaming.connectors.elasticsearch2.ElasticsearchSink.open(ElasticsearchSink.java:164)
        at
org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:38)
        at
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:91)
        at
org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:376)
        at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:256)
        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
        at java.lang.Thread.run(Thread.java:745)

I've installed flink on an EMR cluster from the binary distribution
flink-1.1.1-bin-hadoop27-scala_2.10.tgzand the jar file that is executed
on the cluster is build with mvn clean package(I've attached the pom.xml
for reference).

There is a thread on this list that seems to be related, but I'm afraid
I couldn't draw any conclusions from it:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/classpath-issue-on-yarn-tt6442.html#none

Any idea, what's wrong?

Thanks,
Steffen


Mime
View raw message