flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hung <unicorn.bana...@gmail.com>
Subject Re: How does BucketingSink generate a SUCCESS file when a directory is finished
Date Wed, 31 Jan 2018 09:58:39 GMT
it depends on how you partition your file. in my case I write file per hour,
so I'm sure that file is ready after that hour period, in processing time.
Here, read to be ready means this file contains all the data in that hour
period.

If the downstream runs in a batch way, you may want to ensure the file is
ready.
In this case, ready to read can mean all the data before watermark as
arrived.
You could take the BucketingSink and implement this logic there, maybe wait
until watermark
reaches

Best,

Sendoh



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Mime
View raw message