flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Niels Basjes <Ni...@basjes.nl>
Subject Re: Streaming file source?
Date Fri, 20 Jan 2017 12:21:46 GMT
Thanks!

This sounds really close to what I had in mind.
I'll use this first and see how far I get.

Niels

On Fri, Jan 20, 2017 at 11:27 AM, Stephan Ewen <sewen@apache.org> wrote:

> Hi Niels!
>
> There is the Continuous File Monitoring Source, used via
>
> StreamExecutionEnvironment.readFile(FileInputFormat<OUT>
> inputFormat, String filePath, FileProcessingMode watchType, long interval);
>
> This can be used to both continuously ingest from files, or to read files
> once.
>
> Kostas can probably comment more about whether and how you can make the
> file order deterministic.
>
> Stephan
>
>
> On Fri, Jan 20, 2017 at 11:20 AM, Niels Basjes <Niels@basjes.nl> wrote:
>
>> Hi,
>>
>> For testing and optimizing a streaming application I want to have a "100%
>> accurate repeatable" substitute for a Kafka source.
>> I was thinking of creating a streaming source class that simply reads the
>> records from a (static unchanging) set of files.
>> Each file would then produce the data which (in the live situation) come
>> from a single Kafka partition.
>>
>> I hate reinventing the wheel so I'm wondering is something like this
>> already been built by someone?
>> If so, where can I find it?
>>
>> --
>> Best regards / Met vriendelijke groeten,
>>
>> Niels Basjes
>>
>
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Mime
View raw message