flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lohith Samaga M <Lohith.Sam...@mphasis.com>
Subject Re: How to share text file across tasks at run time in flink.
Date Tue, 23 Aug 2016 13:45:47 GMT
May be you could use Cassandra to store and fetch all such reference data.  This way the reference
data can be updated without restarting your application.


Sent from my Sony Xperia™ smartphone

---- Baswaraj Kasture wrote ----

Thanks Kostas !
I am using DataStream API.

I have few config/property files (key vale text file) and also have business rule files (json).
These rules and configurations are needed when we process incoming event.
Is there any way to share them to task nodes from driver program ?
I think this is very common use case and am sure other users may face similar issues.


On Mon, Aug 22, 2016 at 4:56 PM, Kostas Kloudas <k.kloudas@data-artisans.com<mailto:k.kloudas@data-artisans.com>>
Hello Baswaraj,

Are you using the DataSet (batch) or the DataStream API?

If you are in the first, you can use a broadcast variable<https://ci.apache.org/projects/flink/flink-docs-master/apis/batch/index.html#broadcast-variables>
for your task.
If you are using the DataStream one, then there is no proper support for that.


On Aug 20, 2016, at 12:33 PM, Baswaraj Kasture <kbaswaraj1@gmail.com<mailto:kbaswaraj1@gmail.com>>

Am running Flink standalone cluster.

I have text file that need to be shared across tasks when i submit my application.
in other words , put this text file in class path of running tasks.

How can we achieve this with flink ?

In spark, spark-submit has --jars option that puts all the files specified in class path of
executors (executors run in separate JVM and spawned dynamically, so it is possible).

Flink's task managers run tasks in separate thread under taskmanager JVM (?) , how can we
make this text file to be accessible on all tasks spawned by current application ?

Using HDFS, NFS or including file in program jar is one way that i know, but am looking for
solution that can allows me to provide text file at run time and still accessible in all tasks.

Information transmitted by this e-mail is proprietary to Mphasis, its associated companies
and/ or its customers and is intended 
for use only by the individual or entity to which it is addressed, and may contain information
that is privileged, confidential or 
exempt from disclosure under applicable law. If you are not the intended recipient or it appears
that this mail has been forwarded 
to you without proper authority, you are notified that any use or dissemination of this information
in any manner is strictly 
prohibited. In such cases, please notify us immediately at mailmaster@mphasis.com and delete
this mail from your records.

View raw message