flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: Flink 0.10.2 and Kafka 0.8.1
Date Mon, 18 Apr 2016 14:07:47 GMT
If you want you can share the pom of your project privately.

On Mon, Apr 18, 2016 at 4:05 PM, Robert Schmidtke <ro.schmidtke@gmail.com>
wrote:

> You're right, it does not. When including it the resulting jar has the
> Kafka dependencies bundled. Now it's up to me to figure out the difference
> between the sample project and the one I'm working on.
>
> Thanks! Really quick help.
>
> Robert
>
> On Mon, Apr 18, 2016 at 4:02 PM, Robert Metzger <rmetzger@apache.org>
> wrote:
>
>> Hi,
>> the problem with the posted project is that it doesn't have the Flink
>> kafka connector as a dependency.
>>
>> On Mon, Apr 18, 2016 at 3:56 PM, Robert Schmidtke <ro.schmidtke@gmail.com
>> > wrote:
>>
>>> Hi Robert,
>>>
>>> thanks for your hints. I was not sure whether I was building a proper
>>> fat jar, as I have not used the Flink Archetype for my project. However, I
>>> have set up a sample project at
>>>
>>> https://github.com/robert-schmidtke/flink-test/
>>>
>>> which is nothing more than the Quickstart Archetype plus the
>>> instructions to bundle the Kafka dependencies. The resulting jars (mvn
>>> clean package -Pbuild-jar and mvn clean package) do not contain the
>>> org/apache/kafka/** classes. Can you have a quick look at the pom? However,
>>> as I said, it's verbatim Archetype+Flink Docs.
>>>
>>> Thanks a lot in advance!
>>>
>>> Robert
>>>
>>>
>>>
>>> On Mon, Apr 18, 2016 at 12:36 PM, Robert Metzger <rmetzger@apache.org>
>>> wrote:
>>>
>>>> Hi,
>>>> did you check your user jar if it contains the Kafka classes?
>>>> Are you building a fat jar? Are you manually excluding any dependencies?
>>>>
>>>> Flink's 0.10.2 Kafka connector depends on Kafka 0.8.2.0 [1] which in
>>>> turn depends on kafka-clients 0.8.2.0 [2]. And the "kafka-clients"
>>>> dependency also contains the org.apache.kafka.common.Node class (the
>>>> LegacyFetcher needs that class).
>>>>
>>>>
>>>> [1]
>>>> https://repo1.maven.org/maven2/org/apache/flink/flink-connector-kafka/0.10.2/flink-connector-kafka-0.10.2.pom
>>>> [2]
>>>> https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.10/0.8.2.0/kafka_2.10-0.8.2.0.pom
>>>>
>>>> On Mon, Apr 18, 2016 at 7:42 AM, Balaji Rajagopalan <
>>>> balaji.rajagopalan@olacabs.com> wrote:
>>>>
>>>>> I had fought with 0.8.0.2 kafka and flink 0.10.2 scala version 2.11,
>>>>> was never able to get it working confounded with noclassdeffounderror,
>>>>> moved to flink 1.0.0 with kafka 0.8.0.2  scala version 2.11 things worked
>>>>> for me, if moving to flink 1.0.0 is an option for you do so.
>>>>>
>>>>> balaji
>>>>>
>>>>> On Mon, Apr 18, 2016 at 3:19 AM, Robert Schmidtke <
>>>>> ro.schmidtke@gmail.com> wrote:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> I have a Kafka cluster running on version 0.8.1, hence I'm using
the
>>>>>> FlinkKafkaConsumer081. When running my program, I saw a
>>>>>> NoClassDefFoundError for org.apache.kafka.common.Node. So I packaged
my
>>>>>> binaries according to
>>>>>> https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution,
>>>>>> however I'm still seeing the error.
>>>>>>
>>>>>> I played around a bit and it turns out I have to package
>>>>>> kafka-clients v. 0.8.2.0 instead of kafka_2.10 v. 0.8.1 with my program.
Is
>>>>>> there an error in the documentation or have I not figured out something
>>>>>> properly?
>>>>>>
>>>>>> Thanks!
>>>>>> Robert
>>>>>>
>>>>>> --
>>>>>> My GPG Key ID: 336E2680
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> My GPG Key ID: 336E2680
>>>
>>
>>
>
>
> --
> My GPG Key ID: 336E2680
>

Mime
View raw message