flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: yarnship option
Date Mon, 22 May 2017 19:14:39 GMT
Hi,

this issue is unexpected :) Can you double check if the
job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar contains
org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 ?
Also, were there any previous Kafka09 related exceptions in the log??

>From this SO answer, it seems that this is not really the classical
classNotFoundException, but a bit differenT:
https://stackoverflow.com/a/5756989/568695

On Mon, May 22, 2017 at 5:12 PM, Mikhail Pryakhin <m.pryahin@cleverdata.ru>
wrote:

> Hi all!
>
> I'm playing with flink streaming job on yarn cluster. The job consumes
> events from kafka and prints them to the standard out.
> The job uses flink-connector-kafka-0.9_2.11-1.2.1.jar library that is
> passed via the --yarnship option.
> Here is the way I run the job:
>
> export HADOOP_USER_NAME=hdfs; \
> export HADOOP_CONF_DIR=/etc/hadoop/conf/; \
> /opt/flink-1.2.1/bin/flink run \
>         -yst \
>         -yt /home/user/job-libs  \
>         -m yarn-cluster \
>         -yn 3 \
>         -c com.flink.Test \
>         flink-test_2.11-1.0.0-SNAPSHOT.jar
>
> Finally the job fails complaing that it can't find the class:
> java.lang.NoClassDefFoundError: org/apache/flink/streaming/
> connectors/kafka/FlinkKafkaConsumer09
>
> I looked through the jobmanager.log and found that the
> flink-connector-kafka-0.9_2.11-1.2.1.jar library is added to the
> classpath:
>
> 2017-05-22 17:41:59,637 INFO  org.apache.flink.yarn.YarnApplicationMasterRunner
>            -  Classpath: job-libs/flink-connector-
> kafka-0.9_2.11-1.2.1.jar:<another jars...>
>
> Could please help to figure out why the class org/apache/flink/streaming/
> connectors/kafka/FlinkKafkaConsumer09 can't be loaded inspite of the jar
> containing this class was added to the classpath
>
> ----
> Mike Pryakhin

Mime
View raw message