flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Federico D'Ambrosio" <federico.dambro...@smartlab.ws>
Subject Possible conflict between in Flink connectors
Date Fri, 25 Aug 2017 15:36:51 GMT
Hello everyone, I'm new to Flink and am encountering a nasty problem while
trying to submit a streaming Flink Job. I'll try to explain it as
thoroughly as possible.

Premise: I'm using an HDP 2.6 hadoop cluster, with hadoop version
2.7.3.2.6.1.0-129, Flink compiled from sources accordingly (maven 3.0.5) as
per documentation
<https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/building.html#vendor-specific-versions>
and I submit the job using yarn-session.sh and then flink run.

Getting into more details, the Flink Job is a fat jar built with sbt
assembly and the following dependencies:

libraryDependencies += "org.apache.flink" %% "flink-scala" %
flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %% "flink-streaming-scala" %
flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %% "flink-connector-kinesis"
% flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %% "flink-hbase" %
flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %%
"flink-connector-filesystem" % flinkVersion % "provided"
libraryDependencies += "org.apache.hbase" % "hbase-common" % "1.3.0"
libraryDependencies += "org.joda" % "joda-convert" % "1.8.3"
libraryDependencies += "com.typesafe.play" %% "play-json" % "2.6.2"


assemblyOption in assembly := (assemblyOption in
assembly).value.copy(includeScala = false)


inside the flink lib folder I have the following Jars:

flink-connector-filesystem_2.10-1.3.2.jar
flink-python_2.10-1.3.2.jar
flink-connector-kinesis_2.10-1.3.2.jar
flink-shaded-hadoop2-uber-1.3.2.jar
flink-dist_2.10-1.3.2.jar
log4j-1.2.17.jar
flink-hbase_2.10-1.3.2.jar
slf4j-log4j12-1.7.7.jar

Right now, the issue is that when I flink run the job, I get
NoClassDefFound for org/apache/hadoop/hbase/client/Put class, despite it
being inside the fat jar. So, I tried putting both hbase-common.jar and
hbase-client.jars inside the lib folder, getting as a result another
NoClassDefFound, only, this time, for com/google/common/collect/ListMultimap.
Now, I noticed that flink-connector-kinesis and all the hbase packages
(both hbase-common and flink-hbase deps) have a different version of guava
as dependency: the first one needs version 18.0, while the second one needs
12.0.1 version. I, then, tried to enable the shading of guava packages with
the following:

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("com.google.**" -> "shade.com.google.@1").inAll
)


Only to no avail, getting the exact same error as before. And I really
don't know how to submit the job without pouring every single dependency
inside the lib folder. I thought it could be a compatibility issue with the
YARN cluster, but I tried launching the job locally only to get the same
error.

As an additional info, I get the following warnings if I don't specify
flink packages as provided:

[warn]     * org.scalamacros:quasiquotes_2.10:2.1.0 is selected over 2.0.1
[warn]         +- org.typelevel:macro-compat_2.10:1.1.1
(depends on 2.1.0)
[warn]         +- org.apache.flink:flink-scala_2.10:1.3.2
(depends on 2.0.1)
[warn]
[warn]     * org.mortbay.jetty:jetty-util:6.1.26 is selected over 6.1.26.hwx
[warn]         +- org.apache.hbase:hbase-common:1.3.0
(depends on 6.1.26)
[warn]         +- org.apache.flink:flink-shaded-hadoop2:1.3.2
(depends on 6.1.26.hwx)
[warn]
[warn]     * io.netty:netty:3.8.0.Final is selected over 3.7.0.Final
[warn]         +- com.data-artisans:flakka-remote_2.10:2.3-custom
(depends on 3.8.0.Final)
[warn]         +- org.apache.zookeeper:zookeeper:3.4.6
(depends on 3.7.0.Final)
[warn]
[warn]     * com.google.guava:guava:18.0 is selected over 12.0.1
[warn]         +- org.apache.flink:flink-connector-kinesis_2.10:1.3.2
(depends on 18.0)
[warn]         +- org.apache.hbase:hbase-prefix-tree:1.3.0
(depends on 12.0.1)
[warn]         +- org.apache.hbase:hbase-procedure:1.3.0
(depends on 12.0.1)
[warn]         +- org.apache.hbase:hbase-client:1.3.0
(depends on 12.0.1)
[warn]         +- org.apache.hbase:hbase-common:1.3.0
(depends on 12.0.1)
[warn]         +- org.apache.hbase:hbase-server:1.3.0
(depends on 12.0.1)
[warn]
[warn]     * io.netty:netty-all:4.0.27.Final is selected over 4.0.23.Final
[warn]         +- org.apache.flink:flink-runtime_2.10:1.3.2
(depends on 4.0.27.Final)
[warn]         +- org.apache.flink:flink-shaded-hadoop2:1.3.2
(depends on 4.0.27.Final)
[warn]         +- org.apache.hbase:hbase-prefix-tree:1.3.0
(depends on 4.0.23.Final)
[warn]         +- org.apache.hbase:hbase-client:1.3.0
(depends on 4.0.23.Final)
[warn]         +- org.apache.hbase:hbase-server:1.3.0
(depends on 4.0.23.Final)
[warn]
[warn]     * junit:junit:4.12 is selected over 3.8.1
[warn]         +- org.apache.hbase:hbase-protocol:1.3.0
(depends on 4.12)
[warn]         +- org.apache.hbase:hbase-annotations:1.3.0
(depends on 4.12)
[warn]         +- org.apache.hbase:hbase-prefix-tree:1.3.0
(depends on 4.12)
[warn]         +- org.apache.hbase:hbase-procedure:1.3.0
(depends on 4.12)
[warn]         +- org.apache.hbase:hbase-client:1.3.0
(depends on 4.12)
[warn]         +- org.apache.hbase:hbase-common:1.3.0
(depends on 4.12)
[warn]         +- org.apache.hbase:hbase-server:1.3.0
(depends on 4.12)
[warn]         +- jline:jline:0.9.94
(depends on 3.8.1)

And here I am asking for help here.

Thank you very much for your attention,
Kind regards,

Federico D'Ambrosio

Mime
View raw message