flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai" <tzuli...@apache.org>
Subject Re: Kafka 09 consumer does not commit offsets
Date Mon, 09 Jan 2017 15:39:29 GMT
Hi,

Not sure what might be going on here. I’m pretty certain that for FlinkKafkaConsumer09 when
checkpointing is turned off, the internally used KafkaConsumer client will auto commit offsets
back to Kafka at a default interval of 5000ms (the default value for “auto.commit.interval.ms”).

Could you perhaps provide the logs of your job (you can send them to me privately if you prefer
to)?
From the logs we should be able to see if the internal KafkaConsumer client is correctly configured
to auto commit and also check if anything strange is going on.

Also, how are you reading the committed offsets in Kafka? I recall there was a problem with
the 08 consumer that resulted in the Kafka cli not correctly showing committed offsets of
consumer groups.
However, the 08 consumer had this problem only because we had to implement the auto offset
committing ourselves. I don’t think this should be a issue for the 09 consumer, since we’re
solely relying on the Kafka client’s own implementation to do the auto offset committing.

Cheers,
Gordon


On January 9, 2017 at 7:55:33 PM, Timo Walther (twalthr@apache.org) wrote:

I'm not a Kafka expert but maybe Gordon (in CC) knows more.  

Timo  


Am 09/01/17 um 11:51 schrieb Renjie Liu:  
> Hi, all:  
> I'm using flink 1.1.3 and kafka consumer 09. I read its code and it  
> says that the kafka consumer will turn on auto offset commit if  
> checkpoint is not enabled. I've turned off checkpoint and it seems  
> that kafka client is not committing to offsets to kafka? The offset is  
> important for helping us monitoring. Anyone has encountered this before?  
> --  
> Liu, Renjie  
> Software Engineer, MVAD  



Mime
View raw message