flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mikhail Pryakhin <m.prya...@cleverdata.ru>
Subject Re: yarnship option
Date Tue, 23 May 2017 13:18:36 GMT
Hi Robert,

Thanks for your advice but this option doesn’t work for us because our job depends on third
party jars with transitive dependencies to the same jar but of different versions and it hardly
possible to say which version will participate in the fat jar..

Anyway thanks a lot for your help.

—— 
Mike Pryakhin


> On 23 May 2017, at 16:04, Robert Metzger <rmetzger@apache.org> wrote:
> 
> Hi Mike,
> 
> I would recommend you to build a "fat jar" containing your application code and all required
dependencies.
> 
> On Tue, May 23, 2017 at 10:33 AM, Mikhail Pryakhin <m.pryahin@cleverdata.ru <mailto:m.pryahin@cleverdata.ru>>
wrote:
> Hi Robert!
> 
> You were right.. it was my fault, I didn’t copy all required dependencies for the flink-connector-kafka.
That made impossible to load the class.
> 
> Everything works as expected but only if I put required dependencies inside the flink
lib folder, but when dependencies are located in a different folder which is passed in the
yarnship command line option then they seem not to be put on the TM's classpath, and the job
fails with ClassNotFoundException. 
> Is there anything else I need to provide to make libraries in that folder to be on the
TM's classpath?
> 
> Many thanks,
> 
> ——
> 
> Mike Pryakhin
> 
> 
>> On 22 May 2017, at 23:06, Mikhail Pryakhin <m.pryahin@cleverdata.ru <mailto:m.pryahin@cleverdata.ru>>
wrote:
>> 
>> Hi Robert!
>> Thanks a lot for your reply!
>> 
>> >Can you double check if the job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar
contains org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 ?
>> 
>> The jar does contain the class org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.class
(The contents of the jar file is listed below)
>> 
>> > Also, were there any previous Kafka09 related exceptions in the log??    
>> No, it was the very first exception… You can find log file attached. 
>> 
>> 
>> the flink-connector-kafka-0.9_2.11-1.2.1.jar contains the following:
>> 
>> root@host:~# unzip -l job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar
>> Archive:  job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar
>>   Length      Date    Time    Name
>> ---------  ---------- -----   ----
>>         0  2017-04-11 01:59   META-INF/
>>       443  2017-04-11 01:59   META-INF/MANIFEST.MF
>>      1451  2017-04-11 01:59   META-INF/DEPENDENCIES
>>     11358  2017-04-11 01:59   META-INF/LICENSE
>>       182  2017-04-11 01:59   META-INF/NOTICE
>>         0  2017-04-11 01:59   org/
>>         0  2017-04-11 01:59   org/apache/
>>         0  2017-04-11 01:59   org/apache/flink/
>>         0  2017-04-11 01:59   org/apache/flink/streaming/
>>         0  2017-04-11 01:59   org/apache/flink/streaming/connectors/
>>         0  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/
>>     11554  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.class
>>      2603  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSink.class
>>      4463  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer09.class
>>      2693  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/Kafka09TableSource.class
>>         0  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/
>>      1125  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerCallBridge.class
>>       570  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/Handover$WakeupException.class
>>     10268  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread.class
>>       570  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/Handover$ClosedException.class
>>      3018  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/Handover.class
>>     11854  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/Kafka09Fetcher.class
>>      2182  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread$CommitCallback.class
>>       311  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread$1.class
>>      2329  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSource.class
>>         0  2017-04-11 01:59   META-INF/maven/
>>         0  2017-04-11 01:59   META-INF/maven/org.apache.flink/
>>         0  2017-04-11 01:59   META-INF/maven/org.apache.flink/flink-connector-kafka-0.9_2.11/
>>      6039  2017-04-11 01:59   META-INF/maven/org.apache.flink/flink-connector-kafka-0.9_2.11/pom.xml
>>       131  2017-04-11 01:59   META-INF/maven/org.apache.flink/flink-connector-kafka-0.9_2.11/pom.properties
>>         0  2017-04-11 01:59   META-INF/maven/org.apache.flink/force-shading/
>>      3285  2017-04-11 01:59   META-INF/maven/org.apache.flink/force-shading/pom.xml
>>       114  2017-04-11 01:59   META-INF/maven/org.apache.flink/force-shading/pom.properties
>> ---------                     -------
>>     76543                     33 files
>> 
>> 
>> 
>> 
>> <flink-root-client-dmpkit-dev-dn1.log>
>> 
>> 
>> ------
>> Mike Pryakhin
>> 
>> 
>> 
>>> On 22 May 2017, at 22:14, Robert Metzger <rmetzger@apache.org <mailto:rmetzger@apache.org>>
wrote:
>>> 
>>> Hi,
>>> 
>>> this issue is unexpected :) Can you double check if the job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar
contains org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 ?
>>> Also, were there any previous Kafka09 related exceptions in the log??
>>> 
>>> From this SO answer, it seems that this is not really the classical classNotFoundException,
but a bit differenT: https://stackoverflow.com/a/5756989/568695 <https://stackoverflow.com/a/5756989/568695>
>>> 
>>> On Mon, May 22, 2017 at 5:12 PM, Mikhail Pryakhin <m.pryahin@cleverdata.ru
<mailto:m.pryahin@cleverdata.ru>> wrote:
>>> Hi all!
>>> 
>>> I'm playing with flink streaming job on yarn cluster. The job consumes events
from kafka and prints them to the standard out.
>>> The job uses flink-connector-kafka-0.9_2.11-1.2.1.jar library that is passed
via the --yarnship option.
>>> Here is the way I run the job:
>>> 
>>> export HADOOP_USER_NAME=hdfs; \
>>> export HADOOP_CONF_DIR=/etc/hadoop/conf/; \
>>> /opt/flink-1.2.1/bin/flink run \
>>>         -yst \
>>>         -yt /home/user/job-libs  \
>>>         -m yarn-cluster \
>>>         -yn 3 \
>>>         -c com.flink.Test \
>>>         flink-test_2.11-1.0.0-SNAPSHOT.jar
>>> 
>>> Finally the job fails complaing that it can't find the class:
>>> java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09
>>> 
>>> I looked through the jobmanager.log and found that the flink-connector-kafka-0.9_2.11-1.2.1.jar
library is added to the classpath:
>>> 
>>> 2017-05-22 17:41:59,637 INFO  org.apache.flink.yarn.YarnApplicationMasterRunner
            -  Classpath: job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar:<another jars...>
>>> 
>>> Could please help to figure out why the class org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09
can't be loaded inspite of the jar containing this class was added to the classpath
>>> 
>>> ----
>>> Mike Pryakhin
>>> 
>> 
> 
> 


Mime
View raw message