flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Baswaraj Kasture <kbaswar...@gmail.com>
Subject How to share text file across tasks at run time in flink.
Date Sat, 20 Aug 2016 10:33:14 GMT
Am running Flink standalone cluster.

I have text file that need to be shared across tasks when i submit my
application.
in other words , put this text file in class path of running tasks.

How can we achieve this with flink ?

In spark, spark-submit has --jars option that puts all the files specified
in class path of executors (executors run in separate JVM and spawned
dynamically, so it is possible).

Flink's task managers run tasks in separate thread under taskmanager JVM
(?) , how can we make this text file to be accessible on all tasks spawned
by current application ?

Using HDFS, NFS or including file in program jar is one way that i know,
but am looking for solution that can allows me to provide text file at run
time and still accessible in all tasks.
Thanks.

Mime
View raw message