flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kien Truong <duckientru...@gmail.com>
Subject Re: flink kafka consumer lag
Date Sun, 09 Jul 2017 13:09:06 GMT
Hi,

You should setup a metric reporter to collect Flink's metrics.

https://ci.apache.org/projects/flink/flink-docs-release-1.2/monitoring/metrics.html

There's a lot of useful information in the metrics, including the 
consumer lags.

I'm using the Graphite reporter with InfluxDB for storage + Grafana to 
display the metrics,

it's been working great so far.


Regards,

Kien


On 7/9/17 1:16 PM, Karthik Deivasigamani wrote:
> Hi,
>     I'm using Flink 1.2.1 and FlinkKafkaConsumer09 to read data from
> Kafka Server (0.10.1.0). My consumer is able to read and everything
> works fine.
> But when I use the kafka_consumer_groups.sh command to find the lag
> metrics it does not return the lag. Upon reading online I found that
> Flink Kafka Consumer uses the low level API and manages the assignment
> of partition itself.
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-kafka-group-question-td8185.html
>
> What is the best way to monitor my flink kafka consumer's lag? Does
> Flink expose the lag metrics per partition somewhere?
>
> ~
> Karthik


Mime
View raw message