flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wouter Zorgdrager <W.D.Zorgdra...@tudelft.nl>
Subject Re: Flink and Prometheus setup in K8s
Date Wed, 15 May 2019 13:27:42 GMT
Hi all,

To answer my own questions I worked on the following solution:

1) Custom Docker image which pulls the Flink image and moves Prometheus jar
to the correct folder [1, 2].
2) I wrote manifests for Kubernetes with service discovery configuration
for Kubernetes [3]. Besides the 'official' Flink Kubernetes manifests, I
added a TM service which exposes all TM metrics locations so that
Prometheus can scrape it. This means that (re)-scaling Flink TM's are
automatically picked up by Prometheus. The repository also includes a
Grafana setup with a simple dashboard.

I thought this might be useful for other users!

Cheers,
Wouter

[1]: https://github.com/wzorgdrager/flink-k8s/blob/master/docker/Dockerfile
[2]: https://hub.docker.com/r/wzorgdrager/flink-prometheus
[3]: https://github.com/wzorgdrager/flink-k8s

Op ma 13 mei 2019 om 14:16 schreef Wouter Zorgdrager <
W.D.Zorgdrager@tudelft.nl>:

> Hey all,
>
> I'm working on a deployment setup with Flink and Prometheus on Kubernetes.
> I'm running into the following issues:
>
> 1) Is it possible to use the default Flink Docker image [1] and enable the
> Prometheus reporter? Modifying the flink-config.yaml is easy, but somehow
> the Prometheus reporter jar needs to be moved within the image. This is
> easy if use my own Dockerfile (as done here [2]) , but I prefer using the
> official one.
> 2) I can define the jobmanager/taskmanager metric endpoints statically,
> but w.r.t. scaling I prefer to have these resolved/discovered dynamically.
> Did anyone get a working setup on this? I came across this resource for
> YARN [3], is there something similar for Kubernetes? Or are there any other
> ways of configuring Prometheus to pick this up automatically?
>
> Thanks a lot for your help!
>
> Kind regards,
> Wouter
>
> [1]: https://hub.docker.com/_/flink/
> [2]:
> https://github.com/mbode/flink-prometheus-example/blob/master/Dockerfile
> [3]: https://github.com/eastcirclek/flink-service-discovery
>

Mime
View raw message