flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: Flink Kafka Consumer Behaviour
Date Mon, 08 Aug 2016 19:27:57 GMT
Hi Prabhu,

I'm pretty sure that the Kafka 09 consumer commits offsets to Kafka when
checkpointing is turned on.

In the FlinkKafkaConsumerBase.notifyCheckpointComplete(), we call
fetcher.commitSpecificOffsetsToKafka(checkpointOffsets);, which calls
this.consumer.commitSync(offsetsToCommit); in
Kafka09Fetcher.commitSpecificOffsetsToKafka().


On Mon, Aug 8, 2016 at 8:24 PM, vprabhu@gmail.com <vprabhu@gmail.com> wrote:

> Hi Stephan,
>
> The flink kafka 09 connector does not do offset commits to kafka when
> checkpointing is turned on. Is there a way to monitor the offset lag in
> this
> case,
>
> I am turning on a flink job that reads data from kafka (has about a week
> data - around 7 TB) , currently the approximate way that I use is the
> number
> of records read shown in the flink-UI and the last offset in kafka.
>
> Thanks,
> Prabhu
>
>
>
> --
> View this message in context: http://apache-flink-user-
> mailing-list-archive.2336050.n4.nabble.com/Flink-Kafka-Consumer-Behaviour-
> tp8257p8375.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>

Mime
View raw message