Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAGr9p8C0n=2RE=mYTjXSXRS2meh4N9y3_kbBUvrvsoU_tMCGnA@mail.gmail.com>
References: 
 <CAO2XNQ-vRD20CfFtVUHBhXHaa_Hor+J8ycu9T2o5uFwYCQg4Ag@mail.gmail.com>
	<CAKRzxphLqH0JG2yYd3ew41+vA9co2x6_dsDznZnM_c0LjPYvsw@mail.gmail.com>
	<CAGr9p8C0n=2RE=mYTjXSXRS2meh4N9y3_kbBUvrvsoU_tMCGnA@mail.gmail.com>
Date: Mon, 18 Apr 2016 15:56:16 +0200
Message-ID: 
 <CAO2XNQ9FDmOd+cfMZgQCBoxhdHX2Eek0DMV6-x+rBz8yxpXVuQ@mail.gmail.com>
Subject: Re: Flink 0.10.2 and Kafka 0.8.1
From: Robert Schmidtke <ro.schmidtke@gmail.com>
To: user@flink.apache.org
Content-Type: multipart/alternative; boundary=089e01538c546e49d50530c2b984

--089e01538c546e49d50530c2b984
Content-Type: text/plain; charset=UTF-8

Hi Robert,

thanks for your hints. I was not sure whether I was building a proper fat
jar, as I have not used the Flink Archetype for my project. However, I have
set up a sample project at

https://github.com/robert-schmidtke/flink-test/

which is nothing more than the Quickstart Archetype plus the instructions
to bundle the Kafka dependencies. The resulting jars (mvn clean package
-Pbuild-jar and mvn clean package) do not contain the org/apache/kafka/**
classes. Can you have a quick look at the pom? However, as I said, it's
verbatim Archetype+Flink Docs.

Thanks a lot in advance!

Robert


On Mon, Apr 18, 2016 at 12:36 PM, Robert Metzger <rmetzger@apache.org>
wrote:

> Hi,
> did you check your user jar if it contains the Kafka classes?
> Are you building a fat jar? Are you manually excluding any dependencies?
>
> Flink's 0.10.2 Kafka connector depends on Kafka 0.8.2.0 [1] which in turn
> depends on kafka-clients 0.8.2.0 [2]. And the "kafka-clients" dependency
> also contains the org.apache.kafka.common.Node class (the LegacyFetcher
> needs that class).
>
>
> [1]
> https://repo1.maven.org/maven2/org/apache/flink/flink-connector-kafka/0.10.2/flink-connector-kafka-0.10.2.pom
> [2]
> https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.10/0.8.2.0/kafka_2.10-0.8.2.0.pom
>
> On Mon, Apr 18, 2016 at 7:42 AM, Balaji Rajagopalan <
> balaji.rajagopalan@olacabs.com> wrote:
>
>> I had fought with 0.8.0.2 kafka and flink 0.10.2 scala version 2.11, was
>> never able to get it working confounded with noclassdeffounderror, moved to
>> flink 1.0.0 with kafka 0.8.0.2  scala version 2.11 things worked for me, if
>> moving to flink 1.0.0 is an option for you do so.
>>
>> balaji
>>
>> On Mon, Apr 18, 2016 at 3:19 AM, Robert Schmidtke <ro.schmidtke@gmail.com
>> > wrote:
>>
>>> Hi everyone,
>>>
>>> I have a Kafka cluster running on version 0.8.1, hence I'm using the
>>> FlinkKafkaConsumer081. When running my program, I saw a
>>> NoClassDefFoundError for org.apache.kafka.common.Node. So I packaged my
>>> binaries according to
>>> https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution,
>>> however I'm still seeing the error.
>>>
>>> I played around a bit and it turns out I have to package kafka-clients
>>> v. 0.8.2.0 instead of kafka_2.10 v. 0.8.1 with my program. Is there an
>>> error in the documentation or have I not figured out something properly?
>>>
>>> Thanks!
>>> Robert
>>>
>>> --
>>> My GPG Key ID: 336E2680
>>>
>>
>>
>


-- 
My GPG Key ID: 336E2680

--089e01538c546e49d50530c2b984
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Hi Robert,</div><div><br></div><div>thanks for your h=
ints. I was not sure whether I was building a proper fat jar, as I have not=
 used the Flink Archetype for my project. However, I have set up a sample p=
roject at</div><div><br></div><div><a href=3D"https://github.com/robert-sch=
midtke/flink-test/">https://github.com/robert-schmidtke/flink-test/</a></di=
v><div><br></div><div>which is nothing more than the Quickstart Archetype p=
lus the instructions to bundle the Kafka dependencies. The resulting jars (=
mvn clean package -Pbuild-jar and mvn clean package) do not contain the org=
/apache/kafka/** classes. Can you have a quick look at the pom? However, as=
 I said, it&#39;s verbatim Archetype+Flink Docs.</div><div><br></div><div>T=
hanks a lot in advance!</div><div><br></div><div>Robert</div><div><br></div=
><div><br></div><div><br></div><div class=3D"gmail_extra"><div class=3D"gma=
il_quote">On Mon, Apr 18, 2016 at 12:36 PM, Robert Metzger <span dir=3D"ltr=
">&lt;<a href=3D"mailto:rmetzger@apache.org" target=3D"_blank">rmetzger@apa=
che.org</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D=
"margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,2=
04,204);border-left-style:solid;padding-left:1ex"><div dir=3D"ltr">Hi,<div>=
did you check your user jar if it contains the Kafka classes?</div><div>Are=
 you building a fat jar? Are you manually excluding any dependencies?</div>=
<div><br></div>Flink&#39;s 0.10.2 Kafka connector depends on Kafka 0.8.2.0 =
[1] which in turn depends on kafka-clients 0.8.2.0 [2]. And the &quot;kafka=
-clients&quot; dependency also contains the org.apache.kafka.common.Node cl=
ass (the LegacyFetcher needs that class).<div><br><div><br></div><div>[1]=
=C2=A0<a href=3D"https://repo1.maven.org/maven2/org/apache/flink/flink-conn=
ector-kafka/0.10.2/flink-connector-kafka-0.10.2.pom" target=3D"_blank">http=
s://repo1.maven.org/maven2/org/apache/flink/flink-connector-kafka/0.10.2/fl=
ink-connector-kafka-0.10.2.pom</a></div><div>[2]=C2=A0<a href=3D"https://re=
po1.maven.org/maven2/org/apache/kafka/kafka_2.10/0.8.2.0/kafka_2.10-0.8.2.0=
.pom" target=3D"_blank">https://repo1.maven.org/maven2/org/apache/kafka/kaf=
ka_2.10/0.8.2.0/kafka_2.10-0.8.2.0.pom</a></div></div></div><div class=3D""=
><div class=3D"h5"><div class=3D"gmail_extra"><br><div class=3D"gmail_quote=
">On Mon, Apr 18, 2016 at 7:42 AM, Balaji Rajagopalan <span dir=3D"ltr">&lt=
;<a href=3D"mailto:balaji.rajagopalan@olacabs.com" target=3D"_blank">balaji=
.rajagopalan@olacabs.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmai=
l_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;border-lef=
t-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir=
=3D"ltr">I had fought with 0.8.0.2 kafka and flink 0.10.2 scala version 2.1=
1, was never able to get it working confounded with noclassdeffounderror, m=
oved to flink 1.0.0 with kafka 0.8.0.2 =C2=A0scala version 2.11 things work=
ed for me, if moving to flink 1.0.0 is an option for you do so.=C2=A0<span>=
<font color=3D"#888888"><div><br></div><div>balaji=C2=A0</div></font></span=
></div><div><div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">=
On Mon, Apr 18, 2016 at 3:19 AM, Robert Schmidtke <span dir=3D"ltr">&lt;<a =
href=3D"mailto:ro.schmidtke@gmail.com" target=3D"_blank">ro.schmidtke@gmail=
.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"ma=
rgin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,=
204);border-left-style:solid;padding-left:1ex"><div dir=3D"ltr">Hi everyone=
,<div><br></div><div>I have a Kafka cluster running on version 0.8.1, hence=
 I&#39;m using the FlinkKafkaConsumer081. When running my program, I saw a =
NoClassDefFoundError for org.apache.kafka.common.Node. So I packaged my bin=
aries according to=C2=A0<a href=3D"https://ci.apache.org/projects/flink/fli=
nk-docs-release-0.10/apis/cluster_execution.html#linking-with-modules-not-c=
ontained-in-the-binary-distribution" target=3D"_blank">https://ci.apache.or=
g/projects/flink/flink-docs-release-0.10/apis/cluster_execution.html#linkin=
g-with-modules-not-contained-in-the-binary-distribution</a>, however I&#39;=
m still seeing the error.</div><div><br></div><div>I played around a bit an=
d it turns out I have to package kafka-clients v. 0.8.2.0 instead of kafka_=
2.10 v. 0.8.1 with my program. Is there an error in the documentation or ha=
ve I not figured out something properly?</div><div><br></div><div>Thanks!</=
div><span><font color=3D"#888888"><div>Robert</div><div><div><br></div>-- <=
br><div><div dir=3D"ltr">My GPG Key ID: 336E2680<br></div></div>
</div></font></span></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br><br clear=3D"all"><div><br></div>-- <br>=
<div class=3D"gmail_signature"><div dir=3D"ltr">My GPG Key ID: 336E2680<br>=
</div></div>
</div></div>

--089e01538c546e49d50530c2b984--