flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Son Mai <hongson1...@gmail.com>
Subject Reserving Kafka offset in Flink after modifying app
Date Mon, 25 Mar 2019 06:49:49 GMT

I have a topic in Kafka that Flink reads from. I parse and write messages
in this topic to BigQuery using streaming insert in batch of 500 messages
using in CountWindow in Flink.

*Problem*: I want to commit manually only when a batch was written
successfully to Bigquery.

I saw that Flink KafkaConsumer does not use offset committing to Kafka but
uses its own checkpointing. I don't know how Flink checkpointing works and
I'm worried that Flink's checkpointing does not solve my following
- let's say I have a Flink job running and processing a batch of 500
messages of Kafka offset 1000-1500.
- I stopped this job before it saves to BigQuery and makes some
modifications to the program. Savepoints did not work when I tried because
it required the operators code does not change.

What I want is when I start the modified app, it would start every time
from offset 1000-1500 in Kafka because these messages have not been written
to BigQuery.

Is there any way to achieve this in Flink?


View raw message