Return-Path: X-Original-To: apmail-flink-user-archive@minotaur.apache.org Delivered-To: apmail-flink-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0597919C32 for ; Mon, 14 Mar 2016 09:45:50 +0000 (UTC) Received: (qmail 44404 invoked by uid 500); 14 Mar 2016 09:45:49 -0000 Delivered-To: apmail-flink-user-archive@flink.apache.org Received: (qmail 44309 invoked by uid 500); 14 Mar 2016 09:45:49 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 44299 invoked by uid 99); 14 Mar 2016 09:45:49 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2016 09:45:49 +0000 Received: from mail-lb0-f176.google.com (mail-lb0-f176.google.com [209.85.217.176]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 06FB21A0040 for ; Mon, 14 Mar 2016 09:45:49 +0000 (UTC) Received: by mail-lb0-f176.google.com with SMTP id bc4so230849702lbc.2 for ; Mon, 14 Mar 2016 02:45:48 -0700 (PDT) X-Gm-Message-State: AD7BkJJSbpsocmA2oM7b3BH1k5lEngv3g6ctKp6Oj+qQVJNP1yC9y+4RQVn0DrTFgSwcoyQarcjX94pr63IoYw== X-Received: by 10.25.28.80 with SMTP id c77mr7750935lfc.5.1457948747655; Mon, 14 Mar 2016 02:45:47 -0700 (PDT) MIME-Version: 1.0 Received: by 10.112.190.67 with HTTP; Mon, 14 Mar 2016 02:45:27 -0700 (PDT) In-Reply-To: References: From: Robert Metzger Date: Mon, 14 Mar 2016 10:45:27 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: kafka.javaapi.consumer.SimpleConsumer class not found To: "user@flink.apache.org" Content-Type: multipart/alternative; boundary=001a11401eba34752e052dff251e --001a11401eba34752e052dff251e Content-Type: text/plain; charset=UTF-8 Can you send me the full build file to further investigate the issue? On Fri, Mar 11, 2016 at 4:56 PM, Balaji Rajagopalan < balaji.rajagopalan@olacabs.com> wrote: > Robert, > That did not fix it ( using flink and connector same version) . Tried > with scala version 2.11, so will try to see scala 2.10 makes any > difference. > > balaji > > On Fri, Mar 11, 2016 at 8:06 PM, Robert Metzger > wrote: > >> Hi, >> >> you have to use the same version for all dependencies from the >> "org.apache.flink" group. >> >> You said these are the versions you are using: >> >> flink.version = 0.10.2 >> kafka.verison = 0.8.2 >> flink.kafka.connection.verion=0.9.1 >> >> For the connector, you also need to use 0.10.2. >> >> >> >> On Fri, Mar 11, 2016 at 9:56 AM, Balaji Rajagopalan < >> balaji.rajagopalan@olacabs.com> wrote: >> >>> I am tyring to use the flink kafka connector, for this I have specified >>> the kafka connector dependency and created a fat jar since default flink >>> installation does not contain kafka connector jars. I have made sure that >>> flink-streaming-demo-0.1.jar has the >>> kafka.javaapi.consumer.SimpleConsumer.class but still I see the class not >>> found exception. >>> >>> The code for kafka connector in flink. >>> >>> val env = StreamExecutionEnvironment.getExecutionEnvironment >>> val prop:Properties = new Properties() >>> prop.setProperty("zookeeper.connect","somezookeer:2181") >>> prop.setProperty("group.id","some-group") >>> prop.setProperty("bootstrap.servers","somebroker:9092") >>> >>> val stream = env >>> .addSource(new FlinkKafkaConsumer082[String]("location", new SimpleStringSchema, prop)) >>> >>> jar tvf flink-streaming-demo-0.1.jar | grep >>> kafka.javaapi.consumer.SimpleConsumer >>> >>> 5111 Fri Mar 11 14:18:36 UTC 2016 >>> *kafka/javaapi/consumer/SimpleConsumer*.class >>> >>> flink.version = 0.10.2 >>> kafka.verison = 0.8.2 >>> flink.kafka.connection.verion=0.9.1 >>> >>> The command that I use to run the flink program in yarn cluster is >>> below, >>> >>> HADOOP_CONF_DIR=/etc/hadoop/conf /usr/share/flink/bin/flink run -c >>> com.dataartisans.flink_demo.examples.DriverEventConsumer -m yarn-cluster >>> -yn 2 /home/balajirajagopalan/flink-streaming-demo-0.1.jar >>> >>> java.lang.NoClassDefFoundError: kafka/javaapi/consumer/SimpleConsumer >>> >>> at >>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.getPartitionsForTopic(FlinkKafkaConsumer.java:691) >>> >>> at >>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.(FlinkKafkaConsumer.java:281) >>> >>> at >>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer082.(FlinkKafkaConsumer082.java:49) >>> >>> at >>> com.dataartisans.flink_demo.examples.DriverEventConsumer$.main(DriverEventConsumer.scala:53) >>> >>> at >>> com.dataartisans.flink_demo.examples.DriverEventConsumer.main(DriverEventConsumer.scala) >>> >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>> >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> >>> at java.lang.reflect.Method.invoke(Method.java:497) >>> >>> at >>> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:497) >>> >>> at >>> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:395) >>> >>> at org.apache.flink.client.program.Client.runBlocking(Client.java:252) >>> >>> at >>> org.apache.flink.client.CliFrontend.executeProgramBlocking(CliFrontend.java:676) >>> >>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:326) >>> >>> at >>> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:978) >>> >>> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1028) >>> >>> Caused by: java.lang.ClassNotFoundException: >>> kafka.javaapi.consumer.SimpleConsumer >>> >>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >>> >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>> >>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) >>> >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>> >>> ... 16 more >>> >>> >>> Any help appreciated. >>> >>> >>> balaji >>> >> >> > --001a11401eba34752e052dff251e Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Can you send me the full build file to further investigate= the issue?

= On Fri, Mar 11, 2016 at 4:56 PM, Balaji Rajagopalan <<= a href=3D"mailto:balaji.rajagopalan@olacabs.com" target=3D"_blank">balaji.r= ajagopalan@olacabs.com> wrote:
Robert,
=C2=A0 That did not fix it ( using flink a= nd connector same version) . Tried with scala version 2.11, so will try to = see scala 2.10 makes any difference.=C2=A0

balaji=C2=A0

<= div class=3D"gmail_quote">On Fri, Mar 11, 2016 at 8:06 PM, Robert Metzger <= span dir=3D"ltr"><rmetzger@apache.org> wrote:
Hi,

you have to use the same versio= n for all dependencies from the "org.apache.flink" group.

You said these are the versions you are using:

flink.version =3D 0.10= .2
kafka.verison =3D 0.8.2
flink.kafka.connection.verion=3D0.9.1
=

For the connector, you also need to use 0.10.2.


<= div>

On Fri, = Mar 11, 2016 at 9:56 AM, Balaji Rajagopalan <balaji.rajagop= alan@olacabs.com> wrote:
I am tyring to use the flink kafka connector, for this I hav= e specified the kafka connector dependency and created a fat jar since defa= ult flink installation does not contain kafka connector jars. I have made s= ure that flink-streaming-demo-0.1.jar has the kafka.javaapi.consumer.Simple= Consumer.class but still I see the class not found exception.=C2=A0
The code for kafka connector in flink.=C2=A0
val env =3D StreamExecutionEnvironm=
ent.getExecutionEnvironment
val prop:Properties=
 =3D new Propert=
ies()
prop.setProperty("zookeeper.connect","somezookeer:2181")
prop.setProperty(<=
span style=3D"color:rgb(0,128,0);font-weight:bold">"group.id","some-group")
prop.setPr=
operty("bootstrap.=
servers",&q=
uot;somebroker:9092")

val stream =3D env
  .addSource(new FlinkKafkaConsumer082[String]("location", new SimpleStringSchema, prop))
<= /div>

jar tvf flink-streaming-demo-0.1.jar | grep kafka.javaapi.consumer= .SimpleConsumer

=C2=A0 5111 Fri Mar 11 14:18:36 UTC 2016 kafka/jav= aapi/consumer/SimpleConsumer.class

flink.version =3D 0.10.2
kafka.verison =3D 0.8.2
flink.kafka.connection.verion=3D0.9.1

The c= ommand that I use to run the flink program in yarn cluster is below,=C2=A0<= /div>

HADOOP_CONF_DIR=3D/etc/hadoop/conf /usr/share/flink/bin/flink run = -c com.dataartisans.flink_demo.examples.DriverEventConsumer=C2=A0 -m yarn-c= luster -yn 2 /home/balajirajagopalan/flink-streaming-demo-0.1.jar

java.lang.NoClassDefFoundError: kafka/javaapi/consumer/SimpleConsu= mer

at org.apache.flink.streaming.connectors.kafka.Flink= KafkaConsumer.getPartitionsForTopic(FlinkKafkaConsumer.java:691)

at org.apache.flink.streaming.connectors.kafka.Flink= KafkaConsumer.<init>(FlinkKafkaConsumer.java:281)

at org.apache.flink.streaming.connectors.kafka.Flink= KafkaConsumer082.<init>(FlinkKafkaConsumer082.java:49)

at com.dataartisans.flink_demo.examples.DriverEventC= onsumer$.main(DriverEventConsumer.scala:53)

at com.dataartisans.flink_demo.examples.DriverEventC= onsumer.main(DriverEventConsumer.scala)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Nati= ve Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(Nativ= eMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(D= elegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)<= /span>

at org.apache.flink.client.program.PackagedProgram.c= allMainMethod(PackagedProgram.java:497)

at org.apache.flink.client.program.PackagedProgram.i= nvokeInteractiveModeForExecution(PackagedProgram.java:395)

at org.apache.flink.client.program.Client.runBlockin= g(Client.java:252)

at org.apache.flink.client.CliFrontend.executeProgra= mBlocking(CliFrontend.java:676)

at org.apache.flink.client.CliFrontend.run(CliFronte= nd.java:326)

at org.apache.flink.client.CliFrontend.parseParamete= rs(CliFrontend.java:978)

at org.apache.flink.client.CliFrontend.main(CliFront= end.java:1028)

Caused by: java.lang.ClassNotFoundException: kafka.javaapi.consume= r.SimpleConsumer

at java.net.URLClassLoader.findClass(URLClassLoader.= java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:= 424)

at sun.misc.Launcher$AppClassLoader.loadClass(Launch= er.java:331)

at java.lang.ClassLoader.loadClass(ClassLoader.java:= 357)

... 16 more


Any help appreciated.=C2=A0

<= span>

balaji=C2=A0




--001a11401eba34752e052dff251e--