flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kostas Kloudas <k.klou...@data-artisans.com>
Subject Re: Writing a reliable Flink source for a NON-replay-able queue/protocol that supports message ACKs
Date Tue, 25 Apr 2017 08:23:41 GMT
Hi Martin! 

For an example of a source that acknowledges received messages, you could check the MessageAcknowledgingSourceBase

and the MultipleIdsMessageAcknowledgingSourceBase that ship with Flink. I hope this will give
you some ideas.

Now for the Flink version on top of which to implement your source, I would suggest the Flink
1.3. The reason is that it will 
come out soon (~1 month) and it will include a lot of new features and bug-fixes. Until then,
it may change a bit, but the APIs
that you will be using, will not change. 

So why not going straight for the more future-proof way?


> On Apr 24, 2017, at 11:20 PM, Martin Eden <martineden131@gmail.com> wrote:
> Hi everyone,
> Are there any examples of how to implement a reliable (zero data loss) Flink source reading
from a system that is not replay-able but supports acknowledging messages?
> Or any pointers of how one can achieve this and how Flink can help?
> I imagine it should involve a write ahead log but not yet clear of how to implement it
and how to integrate with the Flink fault tolerance mechanism. Can Flink maintain the write
ahead log for me?
> Also, does it make sense to start implementing this in the current stable Flink release
1.2 or is there any advantage in implementing it directly in Flink 1.3 since it is coming
up soon anyway?
> Thanks,
> M

View raw message