flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hawin Jiang <hawin.ji...@gmail.com>
Subject Re: Kafka0.8.2.1 + Flink0.9.0 issue
Date Mon, 22 Jun 2015 16:37:31 GMT
Hi  Marton

I have to add whole pom.xml file or just only plugin as below.
I saw L286 to L296 are not correct information in pom.xml.
Thanks.



<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId
>maven-assembly-plugin</artifactId> <version>2.4</version> <configuration>
<
descriptorRefs> <descriptorRef>jar-with-dependencies</descriptorRef> </
descriptorRefs> </configuration> </plugin>

On Thu, Jun 11, 2015 at 1:43 AM, Márton Balassi <balassi.marton@gmail.com>
wrote:

> As for locally I meant the machine that you use for development to see
> whether this works without parallelism. :-) No need to install stuff on
> your Namenode of course.
> Installing Kafka on a machine and having the Kafka Java dependencies
> available for Flink are two very different things. Try adding the following
> [1] to your maven pom. Then execute 'mvn assembly:assembly', this will
> produce a fat jar suffiexed jar-with-dependencies.jar. You should be able
> to run the example form that.
>
> [1]
> https://github.com/mbalassi/flink-dataflow/blob/master/pom.xml#L286-296
>
> On Thu, Jun 11, 2015 at 10:32 AM, Hawin Jiang <hawin.jiang@gmail.com>
> wrote:
>
>> Dear Marton
>>
>> What do you meaning for locally Eclipse with 'Run'.
>> Do you want to me to run it on Namenode?
>> But my namenode didn't install Kafka.  I only installed Kafka on my data
>> node servers.
>> Do I need to install or copy Kafka jar on Namenode? Actually, I don't
>> want to install everything on Name node server.
>> Please advise me.
>> Thanks.
>>
>>
>> My Flink and Hadoop cluster info as below.
>>
>> Flink on NameNode
>> Kafka,Zookeeper and FLink slave1 on Datanode1
>> Kafka,Zookeeper ,and Flink slave2 on Datanode2
>> Kafka, Zookeeper and Flink slave3 on Datanode3
>>
>>
>>
>> On Thu, Jun 11, 2015 at 1:16 AM, Márton Balassi <balassi.marton@gmail.com
>> > wrote:
>>
>>> Dear Hawin,
>>>
>>> No problem, I am gald that you are giving our Kafka connector a try. :)
>>> The dependencies listed look good. Can you run the example locally from
>>> Eclipse with 'Run'? I suspect that maybe your Flink cluster does not have
>>> the access to the kafka dependency then.
>>>
>>> As a quick test you could copy the kafka jars to the lib folder of your
>>> Flink distribution on all the machines in your cluster. Everything that is
>>> there goes to the classpath of Flink. Another workaround with be to build a
>>> fat jar for your project containing all the dependencies with 'mvn
>>> assembly:assembly'. Neither of these are beautiful but would help tracking
>>> down the root cause.
>>>
>>> On Thu, Jun 11, 2015 at 10:04 AM, Hawin Jiang <hawin.jiang@gmail.com>
>>> wrote:
>>>
>>>> Dear Marton
>>>>
>>>>
>>>>
>>>> Thanks for supporting again.
>>>>
>>>> I am running these examples at the same project and I am using Eclipse
>>>> IDE to submit it to my Flink cluster.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Here is my dependencies
>>>>
>>>>
>>>> ******************************************************************************
>>>>
>>>> <dependencies>
>>>>
>>>>         <dependency>
>>>>
>>>>             <groupId>*junit*</groupId>
>>>>
>>>>             <artifactId>*junit*</artifactId>
>>>>
>>>>             <version>4.12</version>
>>>>
>>>>             <scope>test</scope>
>>>>
>>>>         </dependency>
>>>>
>>>>         <dependency>
>>>>
>>>>             <groupId>org.apache.flink</groupId>
>>>>
>>>>             <artifactId>*flink*-java</artifactId>
>>>>
>>>>             <version>0.9.0-milestone-1</version>
>>>>
>>>>         </dependency>
>>>>
>>>>         <dependency>
>>>>
>>>>             <groupId>org.apache.flink</groupId>
>>>>
>>>>             <artifactId>*flink*-clients</artifactId>
>>>>
>>>>             <version>0.9.0-milestone-1</version>
>>>>
>>>>         </dependency>
>>>>
>>>>         <dependency>
>>>>
>>>>             <groupId>org.apache.flink</groupId>
>>>>
>>>>             <artifactId>*flink*-streaming-connectors</artifactId>
>>>>
>>>>             <version>0.9.0-milestone-1</version>
>>>>
>>>>         </dependency>
>>>>
>>>>         <dependency>
>>>>
>>>>             <groupId>org.apache.flink</groupId>
>>>>
>>>>             <artifactId>*flink*-streaming-core</artifactId>
>>>>
>>>>             <version>0.9.0-milestone-1</version>
>>>>
>>>>         </dependency>
>>>>
>>>>         <dependency>
>>>>
>>>>             <groupId>org.apache.kafka</groupId>
>>>>
>>>>             <artifactId>kafka_2.10</artifactId>
>>>>
>>>>             <version>0.8.2.1</version>
>>>>
>>>>         </dependency>
>>>>
>>>>         <dependency>
>>>>
>>>>             <groupId>org.apache.kafka</groupId>
>>>>
>>>>             <artifactId>*kafka*-clients</artifactId>
>>>>
>>>>             <version>0.8.2.1</version>
>>>>
>>>>         </dependency>
>>>>
>>>>         <dependency>
>>>>
>>>>             <groupId>org.apache.hadoop</groupId>
>>>>
>>>>             <artifactId>*hadoop*-*hdfs*</artifactId>
>>>>
>>>>             <version>2.6.0</version>
>>>>
>>>>         </dependency>
>>>>
>>>>         <dependency>
>>>>
>>>>             <groupId>org.apache.hadoop</groupId>
>>>>
>>>>             <artifactId>*hadoop*-*auth*</artifactId>
>>>>
>>>>             <version>2.6.0</version>
>>>>
>>>>         </dependency>
>>>>
>>>>         <dependency>
>>>>
>>>>             <groupId>org.apache.hadoop</groupId>
>>>>
>>>>             <artifactId>*hadoop*-common</artifactId>
>>>>
>>>>             <version>2.6.0</version>
>>>>
>>>>         </dependency>
>>>>
>>>>         <dependency>
>>>>
>>>>             <groupId>org.apache.hadoop</groupId>
>>>>
>>>>             <artifactId>*hadoop*-core</artifactId>
>>>>
>>>>             <version>1.2.1</version>
>>>>
>>>>         </dependency>
>>>>
>>>>     </dependencies>
>>>>
>>>>
>>>>
>>>>
>>>> *****************************************************************************************
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Best regards
>>>>
>>>> Email: hawin.jiang@gmail.com
>>>>
>>>>
>>>>
>>>> *From:* Márton Balassi [mailto:balassi.marton@gmail.com]
>>>> *Sent:* Thursday, June 11, 2015 12:58 AM
>>>> *To:* user@flink.apache.org
>>>> *Subject:* Re: Kafka0.8.2.1 + Flink0.9.0 issue
>>>>
>>>>
>>>>
>>>> Dear Hawin,
>>>>
>>>>
>>>>
>>>> This looks like a dependency issue, the java compiler does not find the
>>>> kafka dependency. How are you trying to run this example? Is it from an IDE
>>>> or submitting it to a flink cluster with bin/flink run? How do you define
>>>> your dependencies, do you use maven or sbt for instance?
>>>>
>>>>
>>>>
>>>> Best,
>>>>
>>>>
>>>>
>>>> Marton
>>>>
>>>>
>>>>
>>>> On Thu, Jun 11, 2015 at 9:43 AM, Hawin Jiang <hawin.jiang@gmail.com>
>>>> wrote:
>>>>
>>>>
>>>>
>>>
>>>
>>
>

Mime
View raw message