flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ankitcha <ankitchaudhary...@gmail.com>
Subject Re: Flink Kafka Consumer Behaviour
Date Tue, 04 Oct 2016 03:51:19 GMT
Hi Prabhu, cc Stephan, Robert, 

I was having similar issues where flink Kafka 09 consumer was not committing
offsets to kafka. After digging into JobManager logs, I found that
checkpoints were getting expired before getting completed and hence
"checkpoint completed" message was being ignored. 

I increased the checkpoint interval from default 10 mins to 30 mins to
verify, and then checkpoints were getting finished way before timeout (~12
mins), and then consumer was correctly updating offsets in kafka. 

This seems to be working for us at the moment, and also note this scenarios
normally happens at the start of the job and the consumer group already has
some decent lag. 

So, you might wanna try increasing checkpoint timeouts and check if that
solves the issue. You should look for following in the jobmanager logs 

[Checkpoint Timer] org.apache.flink.runtime.checkpoint.CheckpointCoordinator
- Checkpoint 37 expired before completing. 
[Checkpoint Timer] org.apache.flink.runtime.checkpoint.CheckpointCoordinator
- Triggering checkpoint 38 @ 1474313373634 
[Checkpoint Timer] org.apache.flink.runtime.checkpoint.CheckpointCoordinator
- Checkpoint 38 expired before completing. 
[Checkpoint Timer] org.apache.flink.runtime.checkpoint.CheckpointCoordinator
- Triggering checkpoint 39 @ 1474313973640 

-- 
Ankit 



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Kafka-Consumer-Behaviour-tp8257p9300.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Mime
View raw message