flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Maximilian Michels (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-3565) FlinkKafkaConsumer does not work with Scala 2.11
Date Wed, 02 Mar 2016 17:14:18 GMT

    [ https://issues.apache.org/jira/browse/FLINK-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176007#comment-15176007
] 

Maximilian Michels commented on FLINK-3565:
-------------------------------------------

Thanks to [~till.rohrmann], I've come up with another approach which is also used in other
projects. The idea is to add a dependency to the root pom which is then explicitly shaded
from all modules. That way, the Shade plugin always generates the so called "effective" poms.
Effective poms have their properties resolved. Thus, we can configure the Scala version like
before in the root pom. When Maven packages the Jars, the Shade plugin will ensure that all
poms are made "effective".

I don't know whether we stick to this approach. It is possible to move to another solution
later on. Most importantly, this change is transparent to the user.

In contrast to this, I've created an independent module in our source. So everything is self-contained
and we don't rely on an external Maven module: https://issues.apache.org/jira/browse/SPARK-3812

> FlinkKafkaConsumer does not work with Scala 2.11 
> -------------------------------------------------
>
>                 Key: FLINK-3565
>                 URL: https://issues.apache.org/jira/browse/FLINK-3565
>             Project: Flink
>          Issue Type: Bug
>          Components: Streaming
>    Affects Versions: 1.0.0
>            Reporter: Aljoscha Krettek
>            Assignee: Maximilian Michels
>            Priority: Blocker
>
> Running a program built against Flink_2.11 dependencies fails on a Flink_2.11 cluster
with this exception:
> java.lang.NoClassDefFoundError: scala/collection/GenTraversableOnce$class
> 	at kafka.utils.Pool.<init>(Pool.scala:28)
> 	at kafka.consumer.FetchRequestAndResponseStatsRegistry$.<init>(FetchRequestAndResponseStats.scala:60)
> 	at kafka.consumer.FetchRequestAndResponseStatsRegistry$.<clinit>(FetchRequestAndResponseStats.scala)
> 	at kafka.consumer.SimpleConsumer.<init>(SimpleConsumer.scala:39)
> 	at kafka.javaapi.consumer.SimpleConsumer.<init>(SimpleConsumer.scala:34)
> 	at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer08.getPartitionsForTopic(FlinkKafkaConsumer08.java:518)
> 	at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer08.<init>(FlinkKafkaConsumer08.java:218)
> 	at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer08.<init>(FlinkKafkaConsumer08.java:193)
> 	at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer08.<init>(FlinkKafkaConsumer08.java:160)
> 	at com.dataartisans.querywindow.WindowJob.main(WindowJob.java:93)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:505)
> 	at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403)
> 	at org.apache.flink.client.program.Client.runBlocking(Client.java:248)
> 	at org.apache.flink.client.CliFrontend.executeProgramBlocking(CliFrontend.java:866)
> 	at org.apache.flink.client.CliFrontend.run(CliFrontend.java:333)
> 	at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1189)
> 	at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1239)
> Caused by: java.lang.ClassNotFoundException: scala.collection.GenTraversableOnce$class
> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> 	... 21 more



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message