flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Justin Yan <justin....@remitly.com>
Subject Re: Flink 1.2 Proper Packaging of flink-table with SBT
Date Wed, 08 Mar 2017 03:50:55 GMT
Of course, 15 minutes after I give up and decide to email the mailing list,
I figure it out - my flink App was using the CollectionsEnvironment and not
the proper RemoteEnvironment.

It is still the case, however, that the `flink-table` JAR built by the
standard commands doesn't include the dependencies it requires, and so I'd
be curious to hear what the proper procedure is for linking against
`flink-table` if you want to avoid the bug I highlighted in the
aforementioned JIRA.

Thank you and sorry for the extra noise!

Justin

On Tue, Mar 7, 2017 at 7:21 PM, Justin Yan <justin.yan@remitly.com> wrote:

> Hello!
>
> We are attempting to use the Flink Table API, but are running into a few
> issues.
>
> We initially started with our dependencies looking something like this:
>
> libraryDependencies ++= Seq(
>   "org.apache.flink" %% "flink-scala" % "1.2.0" % "provided",
>   "org.apache.flink" %% "flink-clients" % "1.2.0" % "provided",
>   "org.apache.flink" %% "flink-table" % "1.2.0",
>   Libraries.specs2,
>   ...)
>
> However, this is mildly annoying since flink-table declares dependencies
> on the flink core modules, and thus brings everything in *anyway*.  On
> top of that, we saw this JIRA: https://issues.apache.
> org/jira/browse/FLINK-5227, which we found concerning, so we decided to
> follow the advice - we downloaded and built Flink-1.2 from source (java 7,
> mvn 3.3) using the following (We're also using the Kinesis connector):
>
> tools/change-scala-version.sh 2.11
> mvn clean install -Pinclude-kinesis -DskipTests
> cd flink-dist
> mvn clean install -Pinclude-kinesis -DskipTests
>
> Once this was done, we took the JAR in "/flink-libraries/flink-table/target/"
> and copied it over to the taskManager "/lib" directory.  Finally, we marked
> our `flink-table` dependency as "provided".  Everything compiles, but when
> I try to actually run a simple job, I get the following error at runtime:
>
> java.lang.NoClassDefFoundError: org/codehaus/commons/compiler/
> CompileException
>
> Indeed, when I peek inside of the `flink-table` JAR, I can't find that
> particular package (and similarly, it isn't in the flink-dist JAR either)
>
> $ jar tf flink-table_2.11-1.2.0.jar | grep codehaus
> $
>
> I then attempted to include this library in my user code by adding:
>
> "org.codehaus.janino" % "janino" % "3.0.6",
>
> to my list of dependencies.  When I run a `jar tf myjar.jar | grep
> CompileException` - I see the class. However, when I run my flink
> application in this fat JAR, I *continue to get the same error*, even
> though I am positive this class is included in the fat JAR.  I eventually
> got around this by placing this jar in the `flink/lib` directory, but I am
> very confused as to how this class cannot be found when I have included it
> in the fat JAR that I am submitting with the Flink CLI to a YARN cluster.
> I mostly wanted to mention this in case it is a bug, but mostly to see if
> anyone else has had trouble with the Table API, and if not, if I have
> structured my project incorrectly to cause these troubles.
>
> Thanks!
>
> Justin
>
>
>
>

Mime
View raw message