flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Way to get accumulators values *during* job execution ?
Date Wed, 02 Dec 2015 17:07:12 GMT
Hi Arnaud!

One thing you can do is to periodically retrieve them by querying the
monitoring API:
https://ci.apache.org/projects/flink/flink-docs-release-0.10/internals/monitoring_rest_api.html

A nice approach would be to let the JobManager eagerly publish the
metrics.  I think that Christian Kreizfeld was working on an extension that
periodically publishes the accumulators to some metrics store. Maybe he can
comment on that.

Greetings,
Stephan


On Wed, Dec 2, 2015 at 5:21 PM, LINZ, Arnaud <ALINZ@bouyguestelecom.fr>
wrote:

> Hello,
>
>
>
> I use Grafana/Graphite to monitor my applications. The Flink GUI is really
> nice, but it disappears after the job completes and consequently is not
> suitable to long-term monitoring.
>
>
>
> For batch applications, I simply send the accumulator’s values at the end
> of the job to my Graphite base.
>
> For streaming applications, it’s more complex as the job never ends. It
> would be nice to have a way of getting current accumulator values (like in
> the GUI)  to push it periodically to Graphite in a monitoring thread. Is
> there any API to get the values during execution ?
>
>
>
> Best regards,
>
> Arnaud
>
> ------------------------------
>
> L'intégrité de ce message n'étant pas assurée sur internet, la société
> expéditrice ne peut être tenue responsable de son contenu ni de ses pièces
> jointes. Toute utilisation ou diffusion non autorisée est interdite. Si
> vous n'êtes pas destinataire de ce message, merci de le détruire et
> d'avertir l'expéditeur.
>
> The integrity of this message cannot be guaranteed on the Internet. The
> company that sent this message cannot therefore be held liable for its
> content nor attachments. Any unauthorized use or dissemination is
> prohibited. If you are not the intended recipient of this message, then
> please delete it and notify the sender.
>

Mime
View raw message