flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dominique De Vito <ddv36...@gmail.com>
Subject How to rewind Kafka cursors into a Flink job ?
Date Mon, 27 Mar 2017 16:27:59 GMT
Hi,

Is there a way to rewind Kafka cursors (while using here Kafka as a
consumer) from (inside) a Flink job ?

Use case [plan A]
* The Flink job would listen 1 main "data" topic + 1 secondary "event" topic
* In case of a given event, the Flink job would rewind all Kafka cursors of
the "data" topic, to go back to the latest cursors and retreat data from
there.

Use case-bis [plan A-bis] :
* The Flink job would listen 1 main "data" topic, dealing with data
according to some params
* This Flink job would listen a WS and in case of a given event, the Flink
job would rewind all Kafka cursors of the "data" topic, to go back from the
latest cursors and retreat data from there, according to some new params.

Plan B ;-)
* Listen the events from outside Flink, and in case of an event, stop the
Flink and relaunch it.

So, if someone has any hint about how to rewind for [plan A] and/or [plan
A-bis] => thank you !

Regards,
Dominique

Mime
View raw message