flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Martin Eden <martineden...@gmail.com>
Subject Writing a reliable Flink source for a NON-replay-able queue/protocol that supports message ACKs
Date Mon, 24 Apr 2017 21:20:38 GMT
Hi everyone,

Are there any examples of how to implement a reliable (zero data loss)
Flink source reading from a system that is not replay-able but supports
acknowledging messages?

Or any pointers of how one can achieve this and how Flink can help?

I imagine it should involve a write ahead log but not yet clear of how to
implement it and how to integrate with the Flink fault tolerance mechanism.
Can Flink maintain the write ahead log for me?

Also, does it make sense to start implementing this in the current stable
Flink release 1.2 or is there any advantage in implementing it directly in
Flink 1.3 since it is coming up soon anyway?


View raw message