beam-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Phil Franklin <>
Subject Re: Beam Metrics questions
Date Fri, 30 Nov 2018 16:10:30 GMT
Hi again, Etienne!  You didn’t mention whether spark is reporting committed values now, but
you also didn’t mention opening a ticket concerning the spark output.  Am I right in inferring
that spark does in fact report committed values?


On 2018/11/30 13:57:43, Etienne Chauchot <> wrote: 
> Hi Phil,> 
> Thanks for using MetricsPusher and Beam in general ! > 
> - MetricsHttpSink works that way: it filters out committed metrics from the json output
when committed metrics are not> 
> supported.  I checked, Flink runner still does not support committed metrics. So there
should be no committed metrics> 
> values in the output json.There might be a bug. I'll open a ticket: thx for pointing
out ! You tested on flink and spark> 
> right? and both output committed metrics values right?> 
> - there is no default mechanism to fallback committed metrics values on attempted ones>

> - Apache Flink does no make flink Accumulators available in detached mode, so indeed,
metrics are not available in this> 
> mode.> 
> CCing dev list.> 
> Etienne> 
> Le lundi 26 novembre 2018 à 15:57 -0600, Phil Franklin a écrit :> 
> > All of the discussion I’ve seen says that Flink and Spark only provided attempted
metric values, but when I use> 
> > MetricsHttpSink and look at the JSON it has both attempted and committed values
(albeit, both the same for my simple> 
> > testing).  Has the metrics processing been updated recently, and I’m just missing
the change updates?  Or are the> 
> > committed values being defaulted to the attempted values? > 
> > > 
> > Also, I’ve seen it mentioned that Flink doesn’t report metrics when in detached
mode.  Is this still the case?> 
> > > 
> > > 
> > Thanks for your help!> 
View raw message