flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: Export user metrics with Flink Prometheus endpoint
Date Mon, 18 May 2020 09:16:46 GMT
Now I see what you mean. I think you would have to somehow set up the 
Flink metrics system as a backend for opencensus. Then the metrics would 
be reported to the same system (prometheus) in this case. In Opencensus 
lingo, this would mean using a Flink-based Stats Exporter instead of the 
Prometheus backend directly.

I don't have experience developing opencensus Stats Exporters but 
working off the provided ones should work. Maybe have a look at the 
prometheus exporter [1] and go from there. The entrypoint for a Flink 
exporter would then probably be

     FlinkStatsCollector.createAndRegister(getMetricGroup());

Best,
Aljoscha

On 06.05.20 17:11, Eleanore Jin wrote:
> Hi Aljoscha,
> 
> Thanks for the response, yes prometheus reporter is already enabled and I
> am able to get the flink metrics from prometheus.
> 
> My question is more like, currently I am using opencensus library
> <https://github.com/census-instrumentation/opencensus-java> to collect
> application metrics, so is there an easy way to integrate this metrics with
> flink metrics endpoint.
> 
> Thanks!
> Eleanore
> 
> On Wed, May 6, 2020 at 7:48 AM Aljoscha Krettek <aljoscha@apache.org> wrote:
> 
>> Hi,
>>
>> that should be possible. Did you have a look at the documentation for
>> setting up a Prometheus metrics reporter:
>>
>> https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#prometheus-orgapacheflinkmetricsprometheusprometheusreporter
>>
>> Best,
>> Aljoscha
>>
>> On 06.05.20 03:02, Eleanore Jin wrote:
>>> Hi all,
>>>
>>> I just wonder is it possible to use Flink Metrics endpoint to allow
>>> Prometheus to scrape user defined metrics?
>>>
>>> Context:
>>> In addition to Flink metrics, we also collect some application level
>>> metrics using opencensus. And we run opencensus agent as side car in
>>> kubernetes pod to collect metrics (opencensus agent talks to task manager
>>> container via rpcs)
>>>
>>> The issue with this approach is: it looks like opencensus agent keeps
>>> staled metrics, causing the metrics reporting inaccurate, and this
>> project
>>> is not actively maintained anymore.
>>>
>>> So I wonder if it is possible to use Flink metrics endpoint for user
>>> defined metrics.
>>>
>>> Thanks a lot!
>>> Eleanore
>>>
>>
>>
> 


Mime
View raw message