flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aitozi <gjying1...@gmail.com>
Subject Re: Telling if a job has caught up with Kafka
Date Mon, 30 Oct 2017 11:59:53 GMT
Hi, rmetzger0

Sorry to reply to this old question, i found that we use the kafka client
0.9 in class kafkaThread which lead to the lose of many other detail metrics
add in kafka client 10 like per partition consumer lag mentioned by this
issuse https://issues.apache.org/jira/browse/FLINK-7945.  i will try to add
the detail metrics 

rmetzger0 wrote
> Sorry for joining this discussion late, but there is already a metric for
> the offset lag in our 0.9+ consumers.
> Its called the "records-lag-max":
> https://kafka.apache.org/documentation/#new_consumer_fetch_monitoring and
> its exposed via Flink's metrics system.
> The only issue is that it only shows the maximum lag across all
> partitions,
> not detailed per-partition metrics.

Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

View raw message