flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Lam <paullin3...@gmail.com>
Subject Re: How to submit a job with dependency jars by flink cli in Flink 1.4.2?
Date Fri, 03 Aug 2018 11:20:40 GMT


> 在 2018年8月3日,19:03,Joshua Fan <joshuafatfan@gmail.com> 写道:
> 
> Hi,
> 
> I'd like to submit a job with dependency jars by flink run, but it failed.
> 
> Here is the script,
> 
> /usr/bin/hadoop/software/flink-1.4.2/bin/flink run \
> -m yarn-cluster -yn 1 -ys 8 -yjm 2148 -ytm 4096 -ynm jarsTest \
> -c StreamExample \
> -C file:/home/work/xxx/lib/commons-math3-3.5.jar \
> -C file:/home/work/xxx/lib/flink-connector-kafka-0.8_2.11-1.4.2.jar \
> ...
> xxx-1.0.jar
> 
> As described in https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/cli.html#usage
, "-C" means to provide the dependency jar.
> 
> After I execute the command, the job succeed to submit, but can not run in flink cluster
on yarn. Exceptions is like below:
> 
> org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot load user class:
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer08
> ClassLoader info: URL ClassLoader:
>     file: '/home/work/xxx/lib/commons-math3-3.5.jar' (missing)
>     file: '/home/work/xxx/lib/flink-connector-kafka-0.8_2.11-1.4.2.jar' (missing)
>     .......
> Class not resolvable through given classloader.
> 	at org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(StreamConfig.java:232)
> 	at org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(OperatorChain.java:95)
> 	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:231)
> 	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> It appears that the two dependency jar cannot be found in TaskManager, so I dig into
the source code, from CliFrontend to PackagedProgram to ClusterClient to JobGraph. It seems
like the dependency jars is put in classpath and userCodeClassLoader in PackagedProgram, but
never upload to the BlobServer in JobGraph where the xxx-1.0.jar is uploaded.
> 
> Am I missing something? In Flink 1.4.2, dependency jar is not supported?
> 
> Hope someone can give me some hint.
> 
> Appreciate it very mush.
> 
> 
> Yours Sincerely
> 
> Joshua
> 
> 
> 

Hi Joshua, 

I think what you’re looking for is `-yt` option, which is used for distributing a specified
directory via YARN to the TaskManager nodes. And you can find its description in the Flink
client by executing `bin/flink`.

> -yt,--yarnship <arg>                 Ship files in the specified directory (t for
transfer)

Best Regards, 
Paul Lam
Mime
View raw message