flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: yarnship option
Date Tue, 23 May 2017 13:04:05 GMT
Hi Mike,

I would recommend you to build a "fat jar" containing your application code
and all required dependencies.

On Tue, May 23, 2017 at 10:33 AM, Mikhail Pryakhin <m.pryahin@cleverdata.ru>
wrote:

> Hi Robert!
>
> You were right.. it was my fault, I didn’t copy all required dependencies
> for the flink-connector-kafka. That made impossible to load the class.
>
> Everything works as expected but only if I put required dependencies
> inside the flink lib folder, but when dependencies are located in a
> different folder which is passed in the yarnship command line option then
> they seem not to be put on the TM's classpath, and the job fails with
> ClassNotFoundException.
> Is there anything else I need to provide to make libraries in that folder
> to be on the TM's classpath?
>
> Many thanks,
>
> ——
>
> Mike Pryakhin
>
>
> On 22 May 2017, at 23:06, Mikhail Pryakhin <m.pryahin@cleverdata.ru>
> wrote:
>
> Hi Robert!
> Thanks a lot for your reply!
>
> >Can you double check if the job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar
> contains org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09
> ?
>
> The jar does contain the class org/apache/flink/streaming/
> connectors/kafka/FlinkKafkaConsumer09.class (The contents of the jar file
> is listed below)
>
> > Also, were there any previous Kafka09 related exceptions in the log??
> No, it was the very first exception… You can find log file attached.
>
>
> the flink-connector-kafka-0.9_2.11-1.2.1.jar contains the following:
>
> root@host:~# unzip -l job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar
> Archive:  job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar
>   Length      Date    Time    Name
> ---------  ---------- -----   ----
>         0  2017-04-11 01:59   META-INF/
>       443  2017-04-11 01:59   META-INF/MANIFEST.MF
>      1451  2017-04-11 01:59   META-INF/DEPENDENCIES
>     11358  2017-04-11 01:59   META-INF/LICENSE
>       182  2017-04-11 01:59   META-INF/NOTICE
>         0  2017-04-11 01:59   org/
>         0  2017-04-11 01:59   org/apache/
>         0  2017-04-11 01:59   org/apache/flink/
>         0  2017-04-11 01:59   org/apache/flink/streaming/
>         0  2017-04-11 01:59   org/apache/flink/streaming/connectors/
>         0  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/
>     11554  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/
> FlinkKafkaConsumer09.class
>      2603  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/
> Kafka09JsonTableSink.class
>      4463  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/
> FlinkKafkaProducer09.class
>      2693  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/
> Kafka09TableSource.class
>         0  2017-04-11 01:59   org/apache/flink/streaming/
> connectors/kafka/internal/
>      1125  2017-04-11 01:59   org/apache/flink/streaming/
> connectors/kafka/internal/KafkaConsumerCallBridge.class
>       570  2017-04-11 01:59   org/apache/flink/streaming/
> connectors/kafka/internal/Handover$WakeupException.class
>     10268  2017-04-11 01:59   org/apache/flink/streaming/
> connectors/kafka/internal/KafkaConsumerThread.class
>       570  2017-04-11 01:59   org/apache/flink/streaming/
> connectors/kafka/internal/Handover$ClosedException.class
>      3018  2017-04-11 01:59   org/apache/flink/streaming/
> connectors/kafka/internal/Handover.class
>     11854  2017-04-11 01:59   org/apache/flink/streaming/
> connectors/kafka/internal/Kafka09Fetcher.class
>      2182  2017-04-11 01:59   org/apache/flink/streaming/
> connectors/kafka/internal/KafkaConsumerThread$CommitCallback.class
>       311  2017-04-11 01:59   org/apache/flink/streaming/
> connectors/kafka/internal/KafkaConsumerThread$1.class
>      2329  2017-04-11 01:59   org/apache/flink/streaming/connectors/kafka/
> Kafka09JsonTableSource.class
>         0  2017-04-11 01:59   META-INF/maven/
>         0  2017-04-11 01:59   META-INF/maven/org.apache.flink/
>         0  2017-04-11 01:59   META-INF/maven/org.apache.
> flink/flink-connector-kafka-0.9_2.11/
>      6039  2017-04-11 01:59   META-INF/maven/org.apache.
> flink/flink-connector-kafka-0.9_2.11/pom.xml
>       131  2017-04-11 01:59   META-INF/maven/org.apache.
> flink/flink-connector-kafka-0.9_2.11/pom.properties
>         0  2017-04-11 01:59   META-INF/maven/org.apache.
> flink/force-shading/
>      3285  2017-04-11 01:59   META-INF/maven/org.apache.
> flink/force-shading/pom.xml
>       114  2017-04-11 01:59   META-INF/maven/org.apache.
> flink/force-shading/pom.properties
> ---------                     -------
>     76543                     33 files
>
>
>
>
> <flink-root-client-dmpkit-dev-dn1.log>
>
>
> ------
> Mike Pryakhin
>
>
>
> On 22 May 2017, at 22:14, Robert Metzger <rmetzger@apache.org> wrote:
>
> Hi,
>
> this issue is unexpected :) Can you double check if the
> job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar contains
> org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 ?
> Also, were there any previous Kafka09 related exceptions in the log??
>
> From this SO answer, it seems that this is not really the classical
> classNotFoundException, but a bit differenT: https://
> stackoverflow.com/a/5756989/568695
>
> On Mon, May 22, 2017 at 5:12 PM, Mikhail Pryakhin <m.pryahin@cleverdata.ru
> > wrote:
>
>> Hi all!
>>
>> I'm playing with flink streaming job on yarn cluster. The job consumes
>> events from kafka and prints them to the standard out.
>> The job uses flink-connector-kafka-0.9_2.11-1.2.1.jar library that is
>> passed via the --yarnship option.
>> Here is the way I run the job:
>>
>> export HADOOP_USER_NAME=hdfs; \
>> export HADOOP_CONF_DIR=/etc/hadoop/conf/; \
>> /opt/flink-1.2.1/bin/flink run \
>>         -yst \
>>         -yt /home/user/job-libs  \
>>         -m yarn-cluster \
>>         -yn 3 \
>>         -c com.flink.Test \
>>         flink-test_2.11-1.0.0-SNAPSHOT.jar
>>
>> Finally the job fails complaing that it can't find the class:
>> java.lang.NoClassDefFoundError: org/apache/flink/streaming/con
>> nectors/kafka/FlinkKafkaConsumer09
>>
>> I looked through the jobmanager.log and found that the
>> flink-connector-kafka-0.9_2.11-1.2.1.jar library is added to the
>> classpath:
>>
>> 2017-05-22 17:41:59,637 INFO  org.apache.flink.yarn.YarnApplicationMasterRunner
>>            -  Classpath: job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar:<another
>> jars...>
>>
>> Could please help to figure out why the class
>> org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 can't
>> be loaded inspite of the jar containing this class was added to the
>> classpath
>>
>> ----
>> Mike Pryakhin
>
>
>
>
>

Mime
View raw message