flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Henri Heiskanen <henri.heiska...@gmail.com>
Subject Re: Kafka 09 consumer does not commit offsets
Date Mon, 09 Jan 2017 20:45:56 GMT
Hi,

We had the same problem when running 0.9 consumer against 0.10 Kafka.
Upgrading Flink Kafka connector to 0.10 fixed our issue.

Br,
Henkka

On Mon, Jan 9, 2017 at 5:39 PM, Tzu-Li (Gordon) Tai <tzulitai@apache.org>
wrote:

> Hi,
>
> Not sure what might be going on here. I’m pretty certain that for
> FlinkKafkaConsumer09 when checkpointing is turned off, the internally used
> KafkaConsumer client will auto commit offsets back to Kafka at a default
> interval of 5000ms (the default value for “auto.commit.interval.ms”).
>
> Could you perhaps provide the logs of your job (you can send them to me
> privately if you prefer to)?
> From the logs we should be able to see if the internal KafkaConsumer
> client is correctly configured to auto commit and also check if anything
> strange is going on.
>
> Also, how are you reading the committed offsets in Kafka? I recall there
> was a problem with the 08 consumer that resulted in the Kafka cli not
> correctly showing committed offsets of consumer groups.
> However, the 08 consumer had this problem only because we had to implement
> the auto offset committing ourselves. I don’t think this should be a issue
> for the 09 consumer, since we’re solely relying on the Kafka client’s own
> implementation to do the auto offset committing.
>
> Cheers,
> Gordon
>
>
> On January 9, 2017 at 7:55:33 PM, Timo Walther (twalthr@apache.org) wrote:
>
> I'm not a Kafka expert but maybe Gordon (in CC) knows more.
>
> Timo
>
>
> Am 09/01/17 um 11:51 schrieb Renjie Liu:
> > Hi, all:
> > I'm using flink 1.1.3 and kafka consumer 09. I read its code and it
> > says that the kafka consumer will turn on auto offset commit if
> > checkpoint is not enabled. I've turned off checkpoint and it seems
> > that kafka client is not committing to offsets to kafka? The offset is
> > important for helping us monitoring. Anyone has encountered this before?
> > --
> > Liu, Renjie
> > Software Engineer, MVAD
>
>
>

Mime
View raw message