flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai" <tzuli...@apache.org>
Subject Re: Kafka offset commits
Date Mon, 17 Apr 2017 05:39:45 GMT

The FlinkKafkaConsumer in 1.2 is able to restore from older version state snapshots and bridge
the migration, so there should be no problem in reading the offsets from older state. The
smallest or highest offsets will only be used if the offset no longer exists due to Kafka
data retention settings.

Besides this, there was once fix related to the Kafka 0.8 offsets for Flink 1.2.0 [1]. Shortly
put, before the fix, the committed offsets to ZK was off by 1 (wrt to how Kafka itself defines
the committed offsets).
However, this should not affect the behavior of restoring from offsets in savepoints, so it
should be fine.


[1] https://issues.apache.org/jira/browse/FLINK-4723

On 13 April 2017 at 10:55:40 PM, Gwenhael Pasquiers (gwenhael.pasquiers@ericsson.com) wrote:



We’re going to migrate some applications that consume data from a Kafka 0.8 from Flink 1.0
to Flink 1.2.


We are wondering if the offset commitment system changed between those two versions: is there
a risk that the Flink 1.2-based application will start with no offset (thus will use either
the smallest or highest one) ?

Or can we assume that the Flink 1.2 app will resume its work at the same offset than the Flink
1.0 app stopped (if they use the same consumer group id) ?


View raw message