flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: Export user metrics with Flink Prometheus endpoint
Date Mon, 18 May 2020 09:16:46 GMT
Now I see what you mean. I think you would have to somehow set up the 
Flink metrics system as a backend for opencensus. Then the metrics would 
be reported to the same system (prometheus) in this case. In Opencensus 
lingo, this would mean using a Flink-based Stats Exporter instead of the 
Prometheus backend directly.

I don't have experience developing opencensus Stats Exporters but 
working off the provided ones should work. Maybe have a look at the 
prometheus exporter [1] and go from there. The entrypoint for a Flink 
exporter would then probably be



On 06.05.20 17:11, Eleanore Jin wrote:
> Hi Aljoscha,
> Thanks for the response, yes prometheus reporter is already enabled and I
> am able to get the flink metrics from prometheus.
> My question is more like, currently I am using opencensus library
> <https://github.com/census-instrumentation/opencensus-java> to collect
> application metrics, so is there an easy way to integrate this metrics with
> flink metrics endpoint.
> Thanks!
> Eleanore
> On Wed, May 6, 2020 at 7:48 AM Aljoscha Krettek <aljoscha@apache.org> wrote:
>> Hi,
>> that should be possible. Did you have a look at the documentation for
>> setting up a Prometheus metrics reporter:
>> https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#prometheus-orgapacheflinkmetricsprometheusprometheusreporter
>> Best,
>> Aljoscha
>> On 06.05.20 03:02, Eleanore Jin wrote:
>>> Hi all,
>>> I just wonder is it possible to use Flink Metrics endpoint to allow
>>> Prometheus to scrape user defined metrics?
>>> Context:
>>> In addition to Flink metrics, we also collect some application level
>>> metrics using opencensus. And we run opencensus agent as side car in
>>> kubernetes pod to collect metrics (opencensus agent talks to task manager
>>> container via rpcs)
>>> The issue with this approach is: it looks like opencensus agent keeps
>>> staled metrics, causing the metrics reporting inaccurate, and this
>> project
>>> is not actively maintained anymore.
>>> So I wonder if it is possible to use Flink metrics endpoint for user
>>> defined metrics.
>>> Thanks a lot!
>>> Eleanore

View raw message