flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nirmalya Sengupta <sengupta.nirma...@gmail.com>
Subject Initializing global data
Date Tue, 26 Apr 2016 13:25:24 GMT
Hello Flinksters,

I need to initialize a piece of global data at the beginning of a Flink
application. I use this piece as a READ-ONLY for enhancing the streamed
data that come in at the time of execution. So, this piece of data also has
to be present in all the execution nodes. For readability, let's name this
piece as ReferableData.

The ReferableData is essentially a Map[String, UserDefinedClass]. The
source is a regular CSV file, present in the local file system (no HDFS,for
the time being). So, I read the CSV file in the usual manner, parse it,
create instances of UserDefinedClass and create a Map[asStringK,
asInstanceV] and make this available globally.

What I gather from the documents and discussions in this forum is that one
of the ways to achieve this is to use

env.getConfig().setGlobalJobParameters(parameters)

Where 'parameters' is the Map is the ReferableData that I want to create
from the CSV file.

I have two questions:

a) Is this approach of providing look-up data to all nodes, a good practice?

b) Should I read the input CSV file through the
ExecutionEnvironment.readTextFile or using standard IO routines of Java?
Does the community have any preference?

-- Nirmalya




-- 
Software Technologist
http://www.linkedin.com/in/nirmalyasengupta
"If you have built castles in the air, your work need not be lost. That is
where they should be.
Now put the foundation under them."


[image: --]

Nirmalya Sengupta
[image: https://]about.me/sengupta.nirmalya
<https://about.me/sengupta.nirmalya?promo=email_sig&utm_source=email_sig&utm_medium=external_link&utm_campaign=chrome_ext>

Mime
View raw message