beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aviem Zur (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (BEAM-1765) Remove Aggregators from Spark runner
Date Tue, 21 Mar 2017 04:57:42 GMT

    [ https://issues.apache.org/jira/browse/BEAM-1765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15934063#comment-15934063
] 

Aviem Zur edited comment on BEAM-1765 at 3/21/17 4:57 AM:
----------------------------------------------------------

Awesome to see we are moving forward on removing aggregators!

How can this be removed from the runners independently from Java SDK?

Most uses of aggregators in Spark runner is to pass these aggregators to SDK methods and constructors
which require them. For example: {{ReduceFnRunner}}.

Another use are the {{PAssert}} success and counter aggregators which allow runners to ensure
that all assertions in tests actually happened and not pass tests that should have failed.

The only other use is the support of aggregators in {{SparkPipelineResult}} which can be removed
independently.


was (Author: aviemzur):
Awesome to hear we are moving forward on removing aggregators!

How can this be removed from the runners independently from Java SDK?

Most uses of aggregators in Spark runner is to pass these aggregators to SDK methods and constructors
which require them. For example: {{ReduceFnRunner}}.

Another use are the {{PAssert}} success and counter aggregators which allow runners to ensure
that all assertions in tests actually happened and not pass tests that should have failed.

The only other use is the support of aggregators in {{SparkPipelineResult}} which can be removed
independently.

> Remove Aggregators from Spark runner
> ------------------------------------
>
>                 Key: BEAM-1765
>                 URL: https://issues.apache.org/jira/browse/BEAM-1765
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-spark
>            Reporter: Pablo Estrada
>            Assignee: Amit Sela
>
> I have started removing aggregators from the Java SDK, but runners use them in different
ways that I can't figure out well. This is to track the independent effort in Spark.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message