crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gabriel Reid (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CRUNCH-286) ability to specify a different function for combiner & reducer
Date Sat, 26 Oct 2013 05:02:31 GMT

    [ https://issues.apache.org/jira/browse/CRUNCH-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13806002#comment-13806002
] 

Gabriel Reid commented on CRUNCH-286:
-------------------------------------

The contract for a Combiner is that it can be run 0-n times (although in reality I think it's
always run either once or twice) and it must be distributive, so a combiner that is doing
something like adding 17 and is expected to run exactly once would be breaking that contract
anyhow. 

I'm pretty sure that with CombineFns that follow this contract it should always be safe to
leave out the map-phase combine actions.

> ability to specify a different function for combiner & reducer
> --------------------------------------------------------------
>
>                 Key: CRUNCH-286
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-286
>             Project: Crunch
>          Issue Type: New Feature
>          Components: Core
>            Reporter: Stefan De Smit
>            Assignee: Josh Wills
>            Priority: Minor
>         Attachments: 0001-add-combineValues-method-with-2-function-arguments.patch, 0002-.patch,
CRUNCH-286-prototype.patch
>
>
> Extend PGroupedTable with an extra combineValues function that accepts 2 functions: 1
for combiner phase, 1 for reducer phase.
> This way, different algorithm can be applied.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message