beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <>
Subject [jira] [Work logged] (BEAM-5246) Beam metrics exported as flink metrics are not correct
Date Wed, 12 Sep 2018 15:21:00 GMT


ASF GitHub Bot logged work on BEAM-5246:

                Author: ASF GitHub Bot
            Created on: 12/Sep/18 15:20
            Start Date: 12/Sep/18 15:20
    Worklog Time Spent: 10m 
      Work Description: echauchot commented on issue #6319: [BEAM-5246] Metric container should
forward only metrics of the stepName
   @JozoVilcek thanks for the contribution ! And welcome abord !
   @JozoVilcek @mxm `FlinkMetricsContainer` backs a `MetricsContainer` and there is one `MetricsContainer`
per bundle x step. Also `DoFnRunnerWithMetricsUpdate` runs for a particular step to which
the `FlinkMetricsContainer` is scoped to. So I think it is makes a lot of sense. 
   LGTM also.

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:

Issue Time Tracking

    Worklog Id:     (was: 143576)
    Time Spent: 50m  (was: 40m)

> Beam metrics exported as flink metrics are not correct
> ------------------------------------------------------
>                 Key: BEAM-5246
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-flink
>    Affects Versions: 2.6.0
>            Reporter: Jozef Vilcek
>            Assignee: Jozef Vilcek
>            Priority: Major
>             Fix For: 2.7.0
>          Time Spent: 50m
>  Remaining Estimate: 0h
> In Flink UI and fink native MetricReported, I am seeing too many instances of my Beam
metric counter. It looks like the counter is materialised for every operator running within
the task, although is is emitter from only one beam step (which should map to one operator?).
This produces double counting.
> A bit debugging I noticed this is happening for stream jobs. In batch I was not able
to reproduce it. Problem might be in FlinkMetricContainer.
> []
> The update seems to be called from operators after finishing the bundle. Data from accumulator
are flushed to `runtimeContext.getMetricGroup()`. The scope of accumulator seems to be different
than metricGroup as in there with different call the scope components change, especially for
operatorID. It seems like during the run, `metricResult.getStep()` does not match operatorName
of metricGroup where metric is being pushed.

This message was sent by Atlassian JIRA

View raw message