flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai" <tzuli...@apache.org>
Subject Re: Telling if a job has caught up with Kafka
Date Sat, 18 Mar 2017 07:38:55 GMT
the 0.9 / 0.10 version and 0.8 version behave a bit differently right now for the offset committing.

In 0.9 / 0.10, if checkpointing is enabled, the “auto.commit.enable” etc. settings will
be completely ignored and overwritten before used to instantiate the interval Kafka clients,
hence committing will only happen on Flink checkpoints.

In 0.8, this isn’t the case. Both automatic periodic committing and committing on checkpoints
can take place. That’s perhaps why you’re observing the 0.8 consumer to be committing
more frequently.

FYI: This behaviour will be unified in Flink 1.3.0. If you’re interested, you can take a
look at https://github.com/apache/flink/pull/3527.

- Gordon

On March 17, 2017 at 6:07:38 PM, Florian König (florian.koenig@micardo.com) wrote:

Why is that so? The checkpoint contains the Kafka offset and would be able to start reading
wherever it left off, regardless of any offset stored in Kafka or Zookeeper. Why is the offset
not committed regularly, independently from the checkpointing? Or did I misconfigure anything? 
View raw message