flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sugandha Amatya <sugandha.ama...@gmail.com>
Subject Re: Monitoring folder in flink
Date Tue, 24 Oct 2017 12:39:27 GMT
Hi

I found that flink polls directory based on modified date. In windows when
I copy files the modified date remained same. So, PROCESS_CONTINUOUSLY
resolved the issue.

On Tue, Oct 24, 2017 at 6:09 PM, Fabian Hueske <fhueske@gmail.com> wrote:

> Hi,
>
> with PROCESS_CONTINUOUSLY the application monitors the directory and
> processes new arriving files or files that have been modified. In this case
> the application never terminates because it is waiting for new files to
> appear.
> With PROCESS_ONCE, the content of a directory is processed as it was when
> the application was started. Once all files are processed the application
> terminates.
>
> What kind of behavior are you looking for?
>
> Best, Fabian
>
> 2017-10-22 9:37 GMT+02:00 Sugandha Amatya <sugandha.amatya@gmail.com>:
>
>> I have folder where new files arrive at schedule. Why is my flink
>> readfile not reading new files. I have used but *PROCESS_ONCE* and
>> *PROCESS_CONTINUOUSLY*. When I use *PROCESS_CONTINUOUSLY* it reads the same
>> file but the execution does not terminate whereas for PROCESS_ONCE it
>> terminates in IDE.
>>
>>     String path = "C:\\test";
>>
>> StreamExecutionEnvironment env = StreamExecutionEnvironment.get
>> ExecutionEnvironment();
>>
>>     TextInputFormat format = new TextInputFormat(new
>> org.apache.flink.core.fs.Path(path));
>>
>>     DataStream<String> inputStream = env.readFile(format, path,
>> FileProcessingMode.PROCESS_ONCE, 100);
>>
>> Here at stackoverflow
>> <https://stackoverflow.com/questions/46871106/how-to-read-new-files-arriving-in-folder-using-flink>
>> .
>>
>
>

Mime
View raw message