flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dongwon Kim <eastcirc...@gmail.com>
Subject Service discovery for Prometheus on YARN
Date Wed, 01 Aug 2018 09:09:56 GMT
Hi community,

Prior to my presentation [1], I'd like to share a Python script [2] to discover Flink clusters
on YARN and let Prometheus know via its file-based service discovery mechanism [3].

Prometheus needs to pull metrics from Prometheus exporters running inside TaskManagers.
The problem is that, as also discussed in [4], we cannot know in advance hosts and ports to
which Prometheus exporters are bound.
As Chesnay mentioned in [4], unless the information as to which the exporters are bound is
exposed by Flink's REST APIs, the only way is to manually scrape logs of JM and TMs.
Then you need to let Prometheus know where to pull metrics from.

The script is intended to automate the manual process.
Please visit the git repository for the detailed information.


- Dongwon

[1] https://berlin-2018.flink-forward.org/conference-program/#real-time-driving-score-service-using-flink
[2] https://github.com/eastcirclek/flink-service-discovery <https://github.com/eastcirclek/flink-service-discovery>
[3] https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cfile_sd_config%3E
[4] http://mail-archives.apache.org/mod_mbox/flink-user/201801.mbox/%3Cb9b8868a-3026-689a-941f-ac20b42e6612@apache.org%3E
View raw message