flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Schmidtke <ro.schmid...@gmail.com>
Subject Re: Flink 0.10.2 and Kafka 0.8.1
Date Mon, 18 Apr 2016 14:05:51 GMT
You're right, it does not. When including it the resulting jar has the
Kafka dependencies bundled. Now it's up to me to figure out the difference
between the sample project and the one I'm working on.

Thanks! Really quick help.

Robert

On Mon, Apr 18, 2016 at 4:02 PM, Robert Metzger <rmetzger@apache.org> wrote:

> Hi,
> the problem with the posted project is that it doesn't have the Flink
> kafka connector as a dependency.
>
> On Mon, Apr 18, 2016 at 3:56 PM, Robert Schmidtke <ro.schmidtke@gmail.com>
> wrote:
>
>> Hi Robert,
>>
>> thanks for your hints. I was not sure whether I was building a proper fat
>> jar, as I have not used the Flink Archetype for my project. However, I have
>> set up a sample project at
>>
>> https://github.com/robert-schmidtke/flink-test/
>>
>> which is nothing more than the Quickstart Archetype plus the instructions
>> to bundle the Kafka dependencies. The resulting jars (mvn clean package
>> -Pbuild-jar and mvn clean package) do not contain the org/apache/kafka/**
>> classes. Can you have a quick look at the pom? However, as I said, it's
>> verbatim Archetype+Flink Docs.
>>
>> Thanks a lot in advance!
>>
>> Robert
>>
>>
>>
>> On Mon, Apr 18, 2016 at 12:36 PM, Robert Metzger <rmetzger@apache.org>
>> wrote:
>>
>>> Hi,
>>> did you check your user jar if it contains the Kafka classes?
>>> Are you building a fat jar? Are you manually excluding any dependencies?
>>>
>>> Flink's 0.10.2 Kafka connector depends on Kafka 0.8.2.0 [1] which in
>>> turn depends on kafka-clients 0.8.2.0 [2]. And the "kafka-clients"
>>> dependency also contains the org.apache.kafka.common.Node class (the
>>> LegacyFetcher needs that class).
>>>
>>>
>>> [1]
>>> https://repo1.maven.org/maven2/org/apache/flink/flink-connector-kafka/0.10.2/flink-connector-kafka-0.10.2.pom
>>> [2]
>>> https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.10/0.8.2.0/kafka_2.10-0.8.2.0.pom
>>>
>>> On Mon, Apr 18, 2016 at 7:42 AM, Balaji Rajagopalan <
>>> balaji.rajagopalan@olacabs.com> wrote:
>>>
>>>> I had fought with 0.8.0.2 kafka and flink 0.10.2 scala version 2.11,
>>>> was never able to get it working confounded with noclassdeffounderror,
>>>> moved to flink 1.0.0 with kafka 0.8.0.2  scala version 2.11 things worked
>>>> for me, if moving to flink 1.0.0 is an option for you do so.
>>>>
>>>> balaji
>>>>
>>>> On Mon, Apr 18, 2016 at 3:19 AM, Robert Schmidtke <
>>>> ro.schmidtke@gmail.com> wrote:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> I have a Kafka cluster running on version 0.8.1, hence I'm using the
>>>>> FlinkKafkaConsumer081. When running my program, I saw a
>>>>> NoClassDefFoundError for org.apache.kafka.common.Node. So I packaged
my
>>>>> binaries according to
>>>>> https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution,
>>>>> however I'm still seeing the error.
>>>>>
>>>>> I played around a bit and it turns out I have to package kafka-clients
>>>>> v. 0.8.2.0 instead of kafka_2.10 v. 0.8.1 with my program. Is there an
>>>>> error in the documentation or have I not figured out something properly?
>>>>>
>>>>> Thanks!
>>>>> Robert
>>>>>
>>>>> --
>>>>> My GPG Key ID: 336E2680
>>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> My GPG Key ID: 336E2680
>>
>
>


-- 
My GPG Key ID: 336E2680

Mime
View raw message