beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pablo Estrada (JIRA)" <>
Subject [jira] [Assigned] (BEAM-3926) Support MetricsPusher in Dataflow Runner
Date Mon, 26 Mar 2018 18:47:00 GMT


Pablo Estrada reassigned BEAM-3926:

    Assignee: Pablo Estrada  (was: Thomas Groh)

> Support MetricsPusher in Dataflow Runner
> ----------------------------------------
>                 Key: BEAM-3926
>                 URL:
>             Project: Beam
>          Issue Type: Sub-task
>          Components: runner-dataflow
>            Reporter: Scott Wegner
>            Assignee: Pablo Estrada
>            Priority: Major
> See [relevant email thread|].
From [~echauchot]:
> _AFAIK Dataflow being a cloud hosted engine, the related runner is very different from
the others. It just submits a job to the cloud hosted engine. So, no access to metrics container
etc... from the runner. So I think that the MetricsPusher (component responsible for merging
metrics and pushing them to a sink backend) must not be instanciated in DataflowRunner otherwise
it would be more a client (driver) piece of code and we will lose all the interest of being
close to the execution engine (among other things instrumentation of the execution of the
pipelines).  I think that the MetricsPusher needs to be instanciated in the actual Dataflow

This message was sent by Atlassian JIRA

View raw message