flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maciek Próchniak <...@touk.pl>
Subject ContinuousFileMonitoringFunction - deleting file after processing
Date Tue, 18 Oct 2016 07:40:32 GMT

we want to monitor hdfs (or local) directory, read csv files that appear 
and after successful processing - delete them (mainly not to run out of 
disk space...)

I'm not quite sure how to achieve it with current implementation. 
Previously, when we read binary data (unsplittable files) we made small 
hack and deleted them

in our FileInputFormat - but now we want to use splits and detecting 
which split is 'the last one' is no longer so obvious - of course it's 
also problematic when it comes to checkpointing...

So my question is - is there a idiomatic way of deleting processed files?



View raw message