flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Soheil Pourbafrani <soheil.i...@gmail.com>
Subject Flink Load multiple file
Date Mon, 29 Apr 2019 16:32:14 GMT

I want to load multiple file and apply the processing logic on them. After
some searches using the following code I can load all the files in the
directory named "input" into Flink:

TextInputFormat tif = new TextInputFormat(new Path("input"));
DataSet<String> raw = env.readFile(tif, "input//");

If I understood correctly the second / in following of the directory path
will force Flink to load all files in the directory. But I don't know when
creating the TextInputFormat object, why we should pass the path! what is
it's fuctionality?

I also want to know is there a way to pass the path of multiple file
exactly without any regular expression? for example:

DataSet<String> raw = env.readFile(tif, "input/data1, input/data2");

And pls inform me if a better way for loading multiple files exists.


View raw message