flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Márton Balassi <balassi.mar...@gmail.com>
Subject Re: Kafka0.8.2.1 + Flink0.9.0 issue
Date Thu, 11 Jun 2015 08:43:06 GMT
As for locally I meant the machine that you use for development to see
whether this works without parallelism. :-) No need to install stuff on
your Namenode of course.
Installing Kafka on a machine and having the Kafka Java dependencies
available for Flink are two very different things. Try adding the following
[1] to your maven pom. Then execute 'mvn assembly:assembly', this will
produce a fat jar suffiexed jar-with-dependencies.jar. You should be able
to run the example form that.

[1] https://github.com/mbalassi/flink-dataflow/blob/master/pom.xml#L286-296

On Thu, Jun 11, 2015 at 10:32 AM, Hawin Jiang <hawin.jiang@gmail.com> wrote:

> Dear Marton
>
> What do you meaning for locally Eclipse with 'Run'.
> Do you want to me to run it on Namenode?
> But my namenode didn't install Kafka.  I only installed Kafka on my data
> node servers.
> Do I need to install or copy Kafka jar on Namenode? Actually, I don't want
> to install everything on Name node server.
> Please advise me.
> Thanks.
>
>
> My Flink and Hadoop cluster info as below.
>
> Flink on NameNode
> Kafka,Zookeeper and FLink slave1 on Datanode1
> Kafka,Zookeeper ,and Flink slave2 on Datanode2
> Kafka, Zookeeper and Flink slave3 on Datanode3
>
>
>
> On Thu, Jun 11, 2015 at 1:16 AM, Márton Balassi <balassi.marton@gmail.com>
> wrote:
>
>> Dear Hawin,
>>
>> No problem, I am gald that you are giving our Kafka connector a try. :)
>> The dependencies listed look good. Can you run the example locally from
>> Eclipse with 'Run'? I suspect that maybe your Flink cluster does not have
>> the access to the kafka dependency then.
>>
>> As a quick test you could copy the kafka jars to the lib folder of your
>> Flink distribution on all the machines in your cluster. Everything that is
>> there goes to the classpath of Flink. Another workaround with be to build a
>> fat jar for your project containing all the dependencies with 'mvn
>> assembly:assembly'. Neither of these are beautiful but would help tracking
>> down the root cause.
>>
>> On Thu, Jun 11, 2015 at 10:04 AM, Hawin Jiang <hawin.jiang@gmail.com>
>> wrote:
>>
>>> Dear Marton
>>>
>>>
>>>
>>> Thanks for supporting again.
>>>
>>> I am running these examples at the same project and I am using Eclipse
>>> IDE to submit it to my Flink cluster.
>>>
>>>
>>>
>>>
>>>
>>> Here is my dependencies
>>>
>>>
>>> ******************************************************************************
>>>
>>> <dependencies>
>>>
>>>         <dependency>
>>>
>>>             <groupId>*junit*</groupId>
>>>
>>>             <artifactId>*junit*</artifactId>
>>>
>>>             <version>4.12</version>
>>>
>>>             <scope>test</scope>
>>>
>>>         </dependency>
>>>
>>>         <dependency>
>>>
>>>             <groupId>org.apache.flink</groupId>
>>>
>>>             <artifactId>*flink*-java</artifactId>
>>>
>>>             <version>0.9.0-milestone-1</version>
>>>
>>>         </dependency>
>>>
>>>         <dependency>
>>>
>>>             <groupId>org.apache.flink</groupId>
>>>
>>>             <artifactId>*flink*-clients</artifactId>
>>>
>>>             <version>0.9.0-milestone-1</version>
>>>
>>>         </dependency>
>>>
>>>         <dependency>
>>>
>>>             <groupId>org.apache.flink</groupId>
>>>
>>>             <artifactId>*flink*-streaming-connectors</artifactId>
>>>
>>>             <version>0.9.0-milestone-1</version>
>>>
>>>         </dependency>
>>>
>>>         <dependency>
>>>
>>>             <groupId>org.apache.flink</groupId>
>>>
>>>             <artifactId>*flink*-streaming-core</artifactId>
>>>
>>>             <version>0.9.0-milestone-1</version>
>>>
>>>         </dependency>
>>>
>>>         <dependency>
>>>
>>>             <groupId>org.apache.kafka</groupId>
>>>
>>>             <artifactId>kafka_2.10</artifactId>
>>>
>>>             <version>0.8.2.1</version>
>>>
>>>         </dependency>
>>>
>>>         <dependency>
>>>
>>>             <groupId>org.apache.kafka</groupId>
>>>
>>>             <artifactId>*kafka*-clients</artifactId>
>>>
>>>             <version>0.8.2.1</version>
>>>
>>>         </dependency>
>>>
>>>         <dependency>
>>>
>>>             <groupId>org.apache.hadoop</groupId>
>>>
>>>             <artifactId>*hadoop*-*hdfs*</artifactId>
>>>
>>>             <version>2.6.0</version>
>>>
>>>         </dependency>
>>>
>>>         <dependency>
>>>
>>>             <groupId>org.apache.hadoop</groupId>
>>>
>>>             <artifactId>*hadoop*-*auth*</artifactId>
>>>
>>>             <version>2.6.0</version>
>>>
>>>         </dependency>
>>>
>>>         <dependency>
>>>
>>>             <groupId>org.apache.hadoop</groupId>
>>>
>>>             <artifactId>*hadoop*-common</artifactId>
>>>
>>>             <version>2.6.0</version>
>>>
>>>         </dependency>
>>>
>>>         <dependency>
>>>
>>>             <groupId>org.apache.hadoop</groupId>
>>>
>>>             <artifactId>*hadoop*-core</artifactId>
>>>
>>>             <version>1.2.1</version>
>>>
>>>         </dependency>
>>>
>>>     </dependencies>
>>>
>>>
>>>
>>>
>>> *****************************************************************************************
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Best regards
>>>
>>> Email: hawin.jiang@gmail.com
>>>
>>>
>>>
>>> *From:* Márton Balassi [mailto:balassi.marton@gmail.com]
>>> *Sent:* Thursday, June 11, 2015 12:58 AM
>>> *To:* user@flink.apache.org
>>> *Subject:* Re: Kafka0.8.2.1 + Flink0.9.0 issue
>>>
>>>
>>>
>>> Dear Hawin,
>>>
>>>
>>>
>>> This looks like a dependency issue, the java compiler does not find the
>>> kafka dependency. How are you trying to run this example? Is it from an IDE
>>> or submitting it to a flink cluster with bin/flink run? How do you define
>>> your dependencies, do you use maven or sbt for instance?
>>>
>>>
>>>
>>> Best,
>>>
>>>
>>>
>>> Marton
>>>
>>>
>>>
>>> On Thu, Jun 11, 2015 at 9:43 AM, Hawin Jiang <hawin.jiang@gmail.com>
>>> wrote:
>>>
>>>
>>>
>>
>>
>

Mime
View raw message