flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Telco Phone <tel...@yahoo.com>
Subject Processing files
Date Mon, 23 Oct 2017 18:47:16 GMT
Im looking to process files in a directory based on files that are coming in via file transfer.
The files are renamed once the transfer is done to a .DONE.
These are binary files and I need to process billions per day.
What I want to do is process the file and then create a new file called .PROCESSED
I need to have a task thread process a file at a time (unsplitable=true)
The files are in /mnt/DATE/filename.DONE
They are coming in on 4-5 servers at the moment.
I can run a task manager on each host so these will be processed on each server and written
to the same directory.
What is the best way to build a continues list of files to process and hand that filename
to tasks threads running on each host... 
Hope this makes sense... 
Thanks in advance.

View raw message