beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <>
Subject [jira] [Work logged] (BEAM-3310) Push metrics to a backend in an runner agnostic way
Date Tue, 19 Jun 2018 08:10:00 GMT


ASF GitHub Bot logged work on BEAM-3310:

                Author: ASF GitHub Bot
            Created on: 19/Jun/18 08:09
            Start Date: 19/Jun/18 08:09
    Worklog Time Spent: 10m 
      Work Description: echauchot commented on a change in pull request #4548: [BEAM-3310]
Metrics pusher

 File path: runners/flink/src/main/java/org/apache/beam/runners/flink/
 @@ -120,7 +121,9 @@ public PipelineResult run(Pipeline pipeline) {
     if (result instanceof DetachedEnvironment.DetachedJobExecutionResult) {"Pipeline submitted in Detached mode");
-      return new FlinkDetachedRunnerResult();
+      FlinkDetachedRunnerResult flinkDetachedRunnerResult = new FlinkDetachedRunnerResult();
+      // no metricsPusher because metrics are not supported in detached mode
 Review comment:
   @zorro786 in Flink there is no driver like there is in Spark. So user main() code (
and thud MetricsPusher thread) will run inside the Flink client. In default submission mode,
the client lives until the end of pipeline so no issue. In detached mode, the client lives
only the time of submission of the job so, as the Metrics thread is a daemon, it would be
leaking after client is stopped. If we set it not to be a daemon, then it will stop with the
client and no metrics will be pushed. So the current architecture of the MetricsPusher will
not support detached mode in Flink. One of the solutions is to rewrite it all using the DAG
modification but IMHO I find it overkill to change it all just to support one sub-case on
one only runner and more over only metrics pushing and not pulling will be supported. Is there
a strong need for supporting metrics export in detached mode even if regular metrics pulling
(PipelineResult.metrics()) are not supported?

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:

Issue Time Tracking

    Worklog Id:     (was: 113087)
    Time Spent: 13h 40m  (was: 13.5h)

> Push metrics to a backend in an runner agnostic way
> ---------------------------------------------------
>                 Key: BEAM-3310
>                 URL:
>             Project: Beam
>          Issue Type: New Feature
>          Components: runner-extensions-metrics, sdk-java-core
>            Reporter: Etienne Chauchot
>            Assignee: Etienne Chauchot
>            Priority: Major
>          Time Spent: 13h 40m
>  Remaining Estimate: 0h
> The idea is to avoid relying on the runners to provide access to the metrics (either
at the end of the pipeline or while it runs) because they don't have all the same capabilities
towards metrics (e.g. spark runner configures sinks  like csv, graphite or in memory sinks
using the spark engine conf). The target is to push the metrics in the common runner code
so that no matter the chosen runner, a user can get his metrics out of beam.
> Here is the link to the discussion thread on the dev ML:
> And the design doc:

This message was sent by Atlassian JIRA

View raw message