Return-Path: X-Original-To: apmail-flink-user-archive@minotaur.apache.org Delivered-To: apmail-flink-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B1E6F18139 for ; Wed, 2 Dec 2015 17:07:32 +0000 (UTC) Received: (qmail 60037 invoked by uid 500); 2 Dec 2015 17:07:19 -0000 Delivered-To: apmail-flink-user-archive@flink.apache.org Received: (qmail 59950 invoked by uid 500); 2 Dec 2015 17:07:19 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 59940 invoked by uid 99); 2 Dec 2015 17:07:19 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Dec 2015 17:07:19 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 5502E1A5BBE for ; Wed, 2 Dec 2015 17:07:19 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3 X-Spam-Level: *** X-Spam-Status: No, score=3 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.001, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 0pNzQV1cq5-c for ; Wed, 2 Dec 2015 17:07:13 +0000 (UTC) Received: from mail-qg0-f45.google.com (mail-qg0-f45.google.com [209.85.192.45]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id 894BF20FFE for ; Wed, 2 Dec 2015 17:07:13 +0000 (UTC) Received: by qgeb1 with SMTP id b1so38529594qge.1 for ; Wed, 02 Dec 2015 09:07:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type; bh=QAviWVHAJQU0XcFQ6/oJGjpSmJFyqzjxcL+cUX3NV24=; b=AaEd59fvJtBw/v0nl6+ENP46Nu0PlhOBI7HaiXVWhBIkLAeNcFlh+4zbBvlEhW8/uC 1k9Bj4LFOxPHNwgTM347HTgXoTtQ1B6HZBpwDLh12vAbXr2pRgOKYrJP1s1LWzJyzFBu BpZGKDESSfGDWoh6niffw35AaBiIEoJczCApKbhOeewtEL6xc7+z6mLohWjQVHEGWndR xglqV0mdymf/Yr4U6EJfk8RcbX/5oEvGUOzucFAWG6j6riOqCc8TgcA5UYSbVzpPR2zY 2F3sUF6Y5hZuSKQ4GnOjMseqHpt/D40Rgmy+M8YVB+5sVzd4/+394qk/Z6Ah9fqNtx4T 007g== MIME-Version: 1.0 X-Received: by 10.140.226.198 with SMTP id w189mr5548671qhb.83.1449076032667; Wed, 02 Dec 2015 09:07:12 -0800 (PST) Sender: ewenstephan@gmail.com Received: by 10.55.147.1 with HTTP; Wed, 2 Dec 2015 09:07:12 -0800 (PST) In-Reply-To: References: Date: Wed, 2 Dec 2015 18:07:12 +0100 X-Google-Sender-Auth: 4esx59iL2Oqz-Z1qM9eOzyNOXrI Message-ID: Subject: Re: Way to get accumulators values *during* job execution ? From: Stephan Ewen To: user@flink.apache.org Content-Type: multipart/alternative; boundary=001a113710522e05b20525ed4e22 --001a113710522e05b20525ed4e22 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Arnaud! One thing you can do is to periodically retrieve them by querying the monitoring API: https://ci.apache.org/projects/flink/flink-docs-release-0.10/internals/moni= toring_rest_api.html A nice approach would be to let the JobManager eagerly publish the metrics. I think that Christian Kreizfeld was working on an extension that periodically publishes the accumulators to some metrics store. Maybe he can comment on that. Greetings, Stephan On Wed, Dec 2, 2015 at 5:21 PM, LINZ, Arnaud wrote: > Hello, > > > > I use Grafana/Graphite to monitor my applications. The Flink GUI is reall= y > nice, but it disappears after the job completes and consequently is not > suitable to long-term monitoring. > > > > For batch applications, I simply send the accumulator=E2=80=99s values at= the end > of the job to my Graphite base. > > For streaming applications, it=E2=80=99s more complex as the job never en= ds. It > would be nice to have a way of getting current accumulator values (like i= n > the GUI) to push it periodically to Graphite in a monitoring thread. Is > there any API to get the values during execution ? > > > > Best regards, > > Arnaud > > ------------------------------ > > L'int=C3=A9grit=C3=A9 de ce message n'=C3=A9tant pas assur=C3=A9e sur int= ernet, la soci=C3=A9t=C3=A9 > exp=C3=A9ditrice ne peut =C3=AAtre tenue responsable de son contenu ni de= ses pi=C3=A8ces > jointes. Toute utilisation ou diffusion non autoris=C3=A9e est interdite.= Si > vous n'=C3=AAtes pas destinataire de ce message, merci de le d=C3=A9truir= e et > d'avertir l'exp=C3=A9diteur. > > The integrity of this message cannot be guaranteed on the Internet. The > company that sent this message cannot therefore be held liable for its > content nor attachments. Any unauthorized use or dissemination is > prohibited. If you are not the intended recipient of this message, then > please delete it and notify the sender. > --001a113710522e05b20525ed4e22 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Arnaud!

One thing you can do is to p= eriodically retrieve them by querying the monitoring API:=C2=A0https://ci.apache.org/projects/flink/flink-docs-releas= e-0.10/internals/monitoring_rest_api.html

A ni= ce approach would be to let the JobManager eagerly publish the metrics.=C2= =A0 I think that Christian Kreizfeld was working on an extension that perio= dically publishes the accumulators to some metrics store. Maybe he can comm= ent on that.

Greetings,
Stephan


On Wed, Dec 2, 2015 at 5:21 PM, LINZ, Arnaud <ALINZ@bouyguesteleco= m.fr> wrote:

Hello,

=C2=A0

I use Grafana/Graphite= to monitor my applications. The Flink GUI is really nice, but it disappear= s after the job completes and consequently is not suitable to long-term monitoring.

=C2=A0

For batch applications= , I simply send the accumulator=E2=80=99s values at the end of the job to m= y Graphite base.

For streaming applicat= ions, it=E2=80=99s more complex as the job never ends. It would be nice to = have a way of getting current accumulator values (like in the GUI) =C2=A0to push it periodically to Grap= hite in a monitoring thread. Is there any API to get the values during exec= ution ?

=C2=A0

Best regards,

Arnaud



L'int=C3=A9grit=C3=A9 de ce message n'=C3=A9tant pas assur=C3=A9e s= ur internet, la soci=C3=A9t=C3=A9 exp=C3=A9ditrice ne peut =C3=AAtre tenue = responsable de son contenu ni de ses pi=C3=A8ces jointes. Toute utilisation= ou diffusion non autoris=C3=A9e est interdite. Si vous n'=C3=AAtes pas= destinataire de ce message, merci de le d=C3=A9truire et d'avertir l'exp=C3=A9diteur.

The integrity of this message cannot be guaranteed on the Internet. The com= pany that sent this message cannot therefore be held liable for its content= nor attachments. Any unauthorized use or dissemination is prohibited. If y= ou are not the intended recipient of this message, then please delete it and notify the sender.

--001a113710522e05b20525ed4e22--