beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Groh (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (BEAM-2453) The Java DirectRunner should exercise all parts of a CombineFn
Date Fri, 28 Jul 2017 22:27:00 GMT

     [ https://issues.apache.org/jira/browse/BEAM-2453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Thomas Groh resolved BEAM-2453.
-------------------------------
       Resolution: Fixed
    Fix Version/s: 2.2.0

> The Java DirectRunner should exercise all parts of a CombineFn
> --------------------------------------------------------------
>
>                 Key: BEAM-2453
>                 URL: https://issues.apache.org/jira/browse/BEAM-2453
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-direct
>            Reporter: Thomas Groh
>            Assignee: Thomas Groh
>             Fix For: 2.2.0
>
>
> Specifically it should:
> Create some number of accumulators; add elements to these accumulators, merge the created
accumulators, and extract the output.
> This can be performed by replacing the {{Combine.perKey}} composite transform with a
multi-step combine {{CombineBundles -> GroupByKey -> MergeAccumulators}}
> Where {{CombineBundles}} is a {{ParDo}} which takes input {{KV<K, InputT>}} and
produces {{KV<K, AccumT>}}, outputting in {{FinishBundle}} (this can only be performed
if the Combine takes no side inputs or does not have merging windows). {{MergeAccumulators}}
takes in {{KV<K, Iterable<AccumT>>}} and produces {{KV<K, OutputT>}} by
merging all of the accumulators and extracting the output.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message