crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Wills (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CRUNCH-286) ability to specify a different function for combiner & reducer
Date Fri, 25 Oct 2013 21:36:31 GMT

    [ https://issues.apache.org/jira/browse/CRUNCH-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805725#comment-13805725
] 

Josh Wills commented on CRUNCH-286:
-----------------------------------

So you mean treat the map-side phase for the mem pipeline as if there was exactly one observation
per? But that one observation would still be processed by the combiner, so like if the combiner
did something like take the values, sum them, and add 17 to the result, I think that logic
would get executed twice-- once in the combiner and once in the reducer. That seems like a
bad example, now that I look at it-- there may be a more intelligent one that has a similar
issue. :)

> ability to specify a different function for combiner & reducer
> --------------------------------------------------------------
>
>                 Key: CRUNCH-286
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-286
>             Project: Crunch
>          Issue Type: New Feature
>          Components: Core
>            Reporter: Stefan De Smit
>            Assignee: Josh Wills
>            Priority: Minor
>         Attachments: 0001-add-combineValues-method-with-2-function-arguments.patch, 0002-.patch,
CRUNCH-286-prototype.patch
>
>
> Extend PGroupedTable with an extra combineValues function that accepts 2 functions: 1
for combiner phase, 1 for reducer phase.
> This way, different algorithm can be applied.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message