flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: Monitoring folder in flink
Date Tue, 24 Oct 2017 12:24:53 GMT
Hi,

with PROCESS_CONTINUOUSLY the application monitors the directory and
processes new arriving files or files that have been modified. In this case
the application never terminates because it is waiting for new files to
appear.
With PROCESS_ONCE, the content of a directory is processed as it was when
the application was started. Once all files are processed the application
terminates.

What kind of behavior are you looking for?

Best, Fabian

2017-10-22 9:37 GMT+02:00 Sugandha Amatya <sugandha.amatya@gmail.com>:

> I have folder where new files arrive at schedule. Why is my flink readfile
> not reading new files. I have used but *PROCESS_ONCE* and
> *PROCESS_CONTINUOUSLY*. When I use *PROCESS_CONTINUOUSLY* it reads the same
> file but the execution does not terminate whereas for PROCESS_ONCE it
> terminates in IDE.
>
>     String path = "C:\\test";
>
> StreamExecutionEnvironment env = StreamExecutionEnvironment.
> getExecutionEnvironment();
>
>     TextInputFormat format = new TextInputFormat(new
> org.apache.flink.core.fs.Path(path));
>
>     DataStream<String> inputStream = env.readFile(format, path,
> FileProcessingMode.PROCESS_ONCE, 100);
>
> Here at stackoverflow
> <https://stackoverflow.com/questions/46871106/how-to-read-new-files-arriving-in-folder-using-flink>
> .
>

Mime
View raw message