flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hung <unicorn.bana...@gmail.com>
Subject Re: How does BucketingSink generate a SUCCESS file when a directory is finished
Date Wed, 31 Jan 2018 09:58:39 GMT
it depends on how you partition your file. in my case I write file per hour,
so I'm sure that file is ready after that hour period, in processing time.
Here, read to be ready means this file contains all the data in that hour

If the downstream runs in a batch way, you may want to ensure the file is
In this case, ready to read can mean all the data before watermark as
You could take the BucketingSink and implement this logic there, maybe wait
until watermark



Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

View raw message