beam-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Etienne Chauchot <echauc...@apache.org>
Subject Re: Monitoring and Management Tools for Beam an Friends
Date Mon, 26 Mar 2018 14:49:40 GMT
Hi Benjamin
Please know there is an ongoing PR for a runner agnostic metrics feature
https://github.com/apache/beam/pull/4548
Le jeudi 02 mars 2017 à 16:26 +0000, Stas Levin a écrit :
> Hi Benjamin,
> 
> This is somewhat of a hot topic lately, visibility FTW :)
> 
> My experience comes from doing Beam over Spark over Yarn, where we have been using Graphite +
Grafana (4.0+ gives
> alerting as well). 
> 
> In terms of metrics, the Spark runner supports both Spark native metrics (default + Spark
metric sources you
> configure, e.g. JVM), and Beam's, user-defined metrics, all of which are reported to
whatever Spark sink(s) you have
> configured, in our case Graphite.
> 
> There were a few caveats we had to address, most of which had to do with associating
an application's logical name and
> its application id assigned by Yarn. In other words, making all the metrics reported
by a given application id, go
> under its logical name required some effort. However, once it's been done it allows for
dashboards to be built around
> applications' logical names rather than around specific application ids which is very
useful.
> 
> There are more caveats to speak of in the context of metrics and Beam over Spark (and
streaming in particular), so let
> me know if you find the gory details interesting.
> 
> -Stas
> 
> On Thu, Mar 2, 2017 at 5:37 PM Amit Sela <amitsela33@gmail.com> wrote:
> > +Stas Levin 
> > 
> > On Thu, Mar 2, 2017 at 5:30 PM Jean-Baptiste Onofré <jb@nanthrax.net> wrote:
> > > Hi Benjamin,
> > > 
> > > It's a bit related to the Metric discussion on the dev@ mailing list.
> > > 
> > > Today, we leverage the monitoring and management provided by the
> > > execution engine of the runner.
> > > 
> > > For instance, with the Spark runner, we can use the "regular" history
> > > server, codehale reporter, etc. It's also possible to explicitly use a
> > > sink/fn to send data to a backend (Graphite, elasticsearch, ...).
> > > 
> > > We are now discussing about a agnostic way of "pushing" metric to a
> > > backend (something similar to what Karaf Decanter provides).
> > > 
> > > Regards
> > > JB
> > > 
> > > On 03/02/2017 04:13 PM, Stadin, Benjamin wrote:
> > > > Hi,
> > > >
> > > >
> > > >
> > > > I’m trying to collect a list of open source monitoring and management
> > > > tools for Beam and supported runners.
> > > >
> > > >
> > > >
> > > > What do you use in your daily routine, and what’s your experience?
> > > >
> > > >
> > > >
> > > > Regards
> > > >
> > > > Ben
> > > >
> > > >
> > > >
> > > >
> > > >
> > > 
> > > --
> > > Jean-Baptiste Onofré
> > > jbonofre@apache.org
> > > http://blog.nanthrax.net
> > > Talend - http://www.talend.com
> > > 
Mime
View raw message