flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: Flink 0.10.2 and Kafka 0.8.1
Date Mon, 18 Apr 2016 17:43:39 GMT
Yeah, that's the problem
Your Kafka 0.8.1 dependency [1] doesn't depend on kafka-clients. With the
explicit dependency definition, you are overriding the transitive Kafka
dependency from the kafka connector.

[1]
https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.10/0.8.1/kafka_2.10-0.8.1.pom

On Mon, Apr 18, 2016 at 5:38 PM, Robert Schmidtke <ro.schmidtke@gmail.com>
wrote:

> Turns out when I remove the explicit dependency on kafka_2.10 v. 0.8.1,
> then the dependencies are properly included. Guess there was a conflict
> somehow? I'll need to figure out if the rest of the code is fine with
> kafka_2.10 v. 0.8.2.0 as well.
>
> On Mon, Apr 18, 2016 at 4:32 PM, Robert Schmidtke <ro.schmidtke@gmail.com>
> wrote:
>
>> Hi Robert,
>>
>> thanks for your offer. After playing around a bit I would like to take
>> it, if you have the time:
>> https://github.com/robert-schmidtke/HiBench/blob/flink-streaming/src/streambench/flinkbench/pom.xml
>>
>> I would guess the POM is similar to the one in the sample project, yet
>> when building it, the jar does not contain all indirect dependencies.
>>
>> Thanks!!
>>
>> Robert
>>
>> On Mon, Apr 18, 2016 at 4:07 PM, Robert Metzger <rmetzger@apache.org>
>> wrote:
>>
>>> If you want you can share the pom of your project privately.
>>>
>>> On Mon, Apr 18, 2016 at 4:05 PM, Robert Schmidtke <
>>> ro.schmidtke@gmail.com> wrote:
>>>
>>>> You're right, it does not. When including it the resulting jar has the
>>>> Kafka dependencies bundled. Now it's up to me to figure out the difference
>>>> between the sample project and the one I'm working on.
>>>>
>>>> Thanks! Really quick help.
>>>>
>>>> Robert
>>>>
>>>> On Mon, Apr 18, 2016 at 4:02 PM, Robert Metzger <rmetzger@apache.org>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>> the problem with the posted project is that it doesn't have the Flink
>>>>> kafka connector as a dependency.
>>>>>
>>>>> On Mon, Apr 18, 2016 at 3:56 PM, Robert Schmidtke <
>>>>> ro.schmidtke@gmail.com> wrote:
>>>>>
>>>>>> Hi Robert,
>>>>>>
>>>>>> thanks for your hints. I was not sure whether I was building a proper
>>>>>> fat jar, as I have not used the Flink Archetype for my project. However,
I
>>>>>> have set up a sample project at
>>>>>>
>>>>>> https://github.com/robert-schmidtke/flink-test/
>>>>>>
>>>>>> which is nothing more than the Quickstart Archetype plus the
>>>>>> instructions to bundle the Kafka dependencies. The resulting jars
(mvn
>>>>>> clean package -Pbuild-jar and mvn clean package) do not contain the
>>>>>> org/apache/kafka/** classes. Can you have a quick look at the pom?
However,
>>>>>> as I said, it's verbatim Archetype+Flink Docs.
>>>>>>
>>>>>> Thanks a lot in advance!
>>>>>>
>>>>>> Robert
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Apr 18, 2016 at 12:36 PM, Robert Metzger <rmetzger@apache.org
>>>>>> > wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>> did you check your user jar if it contains the Kafka classes?
>>>>>>> Are you building a fat jar? Are you manually excluding any
>>>>>>> dependencies?
>>>>>>>
>>>>>>> Flink's 0.10.2 Kafka connector depends on Kafka 0.8.2.0 [1] which
in
>>>>>>> turn depends on kafka-clients 0.8.2.0 [2]. And the "kafka-clients"
>>>>>>> dependency also contains the org.apache.kafka.common.Node class
(the
>>>>>>> LegacyFetcher needs that class).
>>>>>>>
>>>>>>>
>>>>>>> [1]
>>>>>>> https://repo1.maven.org/maven2/org/apache/flink/flink-connector-kafka/0.10.2/flink-connector-kafka-0.10.2.pom
>>>>>>> [2]
>>>>>>> https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.10/0.8.2.0/kafka_2.10-0.8.2.0.pom
>>>>>>>
>>>>>>> On Mon, Apr 18, 2016 at 7:42 AM, Balaji Rajagopalan <
>>>>>>> balaji.rajagopalan@olacabs.com> wrote:
>>>>>>>
>>>>>>>> I had fought with 0.8.0.2 kafka and flink 0.10.2 scala version
>>>>>>>> 2.11, was never able to get it working confounded with
>>>>>>>> noclassdeffounderror, moved to flink 1.0.0 with kafka 0.8.0.2
 scala
>>>>>>>> version 2.11 things worked for me, if moving to flink 1.0.0
is an option
>>>>>>>> for you do so.
>>>>>>>>
>>>>>>>> balaji
>>>>>>>>
>>>>>>>> On Mon, Apr 18, 2016 at 3:19 AM, Robert Schmidtke <
>>>>>>>> ro.schmidtke@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi everyone,
>>>>>>>>>
>>>>>>>>> I have a Kafka cluster running on version 0.8.1, hence
I'm using
>>>>>>>>> the FlinkKafkaConsumer081. When running my program, I
saw a
>>>>>>>>> NoClassDefFoundError for org.apache.kafka.common.Node.
So I packaged my
>>>>>>>>> binaries according to
>>>>>>>>> https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution,
>>>>>>>>> however I'm still seeing the error.
>>>>>>>>>
>>>>>>>>> I played around a bit and it turns out I have to package
>>>>>>>>> kafka-clients v. 0.8.2.0 instead of kafka_2.10 v. 0.8.1
with my program. Is
>>>>>>>>> there an error in the documentation or have I not figured
out something
>>>>>>>>> properly?
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>> Robert
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> My GPG Key ID: 336E2680
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> My GPG Key ID: 336E2680
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> My GPG Key ID: 336E2680
>>>>
>>>
>>>
>>
>>
>> --
>> My GPG Key ID: 336E2680
>>
>
>
>
> --
> My GPG Key ID: 336E2680
>

Mime
View raw message