flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Justin Yan <justin....@remitly.com>
Subject Flink 1.2 Proper Packaging of flink-table with SBT
Date Wed, 08 Mar 2017 03:21:38 GMT
Hello!

We are attempting to use the Flink Table API, but are running into a few
issues.

We initially started with our dependencies looking something like this:

libraryDependencies ++= Seq(
  "org.apache.flink" %% "flink-scala" % "1.2.0" % "provided",
  "org.apache.flink" %% "flink-clients" % "1.2.0" % "provided",
  "org.apache.flink" %% "flink-table" % "1.2.0",
  Libraries.specs2,
  ...)

However, this is mildly annoying since flink-table declares dependencies on
the flink core modules, and thus brings everything in *anyway*.  On top of
that, we saw this JIRA: https://issues.apache.org/jira/browse/FLINK-5227,
which we found concerning, so we decided to follow the advice - we
downloaded and built Flink-1.2 from source (java 7, mvn 3.3) using the
following (We're also using the Kinesis connector):

tools/change-scala-version.sh 2.11
mvn clean install -Pinclude-kinesis -DskipTests
cd flink-dist
mvn clean install -Pinclude-kinesis -DskipTests

Once this was done, we took the JAR in
"/flink-libraries/flink-table/target/" and copied it over to the
taskManager "/lib" directory.  Finally, we marked our `flink-table`
dependency as "provided".  Everything compiles, but when I try to actually
run a simple job, I get the following error at runtime:

java.lang.NoClassDefFoundError:
org/codehaus/commons/compiler/CompileException

Indeed, when I peek inside of the `flink-table` JAR, I can't find that
particular package (and similarly, it isn't in the flink-dist JAR either)

$ jar tf flink-table_2.11-1.2.0.jar | grep codehaus
$

I then attempted to include this library in my user code by adding:

"org.codehaus.janino" % "janino" % "3.0.6",

to my list of dependencies.  When I run a `jar tf myjar.jar | grep
CompileException` - I see the class. However, when I run my flink
application in this fat JAR, I *continue to get the same error*, even
though I am positive this class is included in the fat JAR.  I eventually
got around this by placing this jar in the `flink/lib` directory, but I am
very confused as to how this class cannot be found when I have included it
in the fat JAR that I am submitting with the Flink CLI to a YARN cluster.
I mostly wanted to mention this in case it is a bug, but mostly to see if
anyone else has had trouble with the Table API, and if not, if I have
structured my project incorrectly to cause these troubles.

Thanks!

Justin

Mime
View raw message