beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Wegner (JIRA)" <j...@apache.org>
Subject [jira] [Created] (BEAM-3926) Support MetricsPusher in Dataflow Runner
Date Mon, 26 Mar 2018 16:56:00 GMT
Scott Wegner created BEAM-3926:
----------------------------------

             Summary: Support MetricsPusher in Dataflow Runner
                 Key: BEAM-3926
                 URL: https://issues.apache.org/jira/browse/BEAM-3926
             Project: Beam
          Issue Type: Sub-task
          Components: runner-dataflow
            Reporter: Scott Wegner
            Assignee: Thomas Groh


See relevant email thread [1]. From [~echauchot]:
 
{noformat}
AFAIK Dataflow being a cloud hosted engine, the related runner is very different from the
others. It just submits a job to the cloud hosted engine. So, no access to metrics container
etc... from the runner. So I think that the MetricsPusher (component responsible for merging
metrics and pushing them to a sink backend) must not be instanciated in DataflowRunner otherwise
it would be more a client (driver) piece of code and we will lose all the interest of being
close to the execution engine (among other things instrumentation of the execution of the
pipelines).  I think that the MetricsPusher needs to be instanciated in the actual Dataflow
engine. {noformat}
 
 

[1 ]https://lists.apache.org/thread.html/2e87f0adcdf8d42317765f298e3e6fdba72917a72d4a12e71e67e4b5@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message