flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Groesbeck <peter.groesb...@gmail.com>
Subject Writing _SUCCESS Files (Streaming and Batch)
Date Mon, 04 May 2020 23:40:20 GMT
I am replacing an M/R job with a Streaming job using the StreamingFileSink
and there is a requirement to generate an empty _SUCCESS file like the old
Hadoop job. I have to implement a similar Batch job to read from backup
files in case of outages or downtime.

The Batch job question was answered here and appears to be still relevant
although if someone could confirm for me that would be great.
https://stackoverflow.com/a/39413810

The question of the Streaming job came up back in 2018 here:
http://mail-archives.apache.org/mod_mbox/flink-user/201802.mbox/%3CFF74EED5-602F-4EAA-9BC1-6CDF56611267@gmail.com%3E

But the solution to use or extend the BucketingSink class seems out of date
now that BucketingSink has been deprecated.

Is there a way to implement a similar solution for StreamingFileSink?

I'm currently on 1.8.1 although I hope to update to 1.10 in the near future.

Thank you,
Peter

Mime
View raw message