crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gabriel Reid (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CRUNCH-345) Force materialization of PCollections prior to multi reduce sorts
Date Sat, 15 Feb 2014 22:22:19 GMT

    [ https://issues.apache.org/jira/browse/CRUNCH-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902550#comment-13902550
] 

Gabriel Reid commented on CRUNCH-345:
-------------------------------------

The HFile-loading tests have the same issue, so they start up a "real" cluster (i.e. not local
jobtracker). It means that the tests run way slower because of all the setup work, but it's
the only way to automatically test stuff like this, so it might not be a bad idea to do the
same thing for the multi-reducer sorting tests.

> Force materialization of PCollections prior to multi reduce sorts
> -----------------------------------------------------------------
>
>                 Key: CRUNCH-345
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-345
>             Project: Crunch
>          Issue Type: Bug
>          Components: MapReduce Patterns
>    Affects Versions: 0.9.0, 0.8.2
>            Reporter: Josh Wills
>         Attachments: CRUNCH-345.patch
>
>
> [~jgmath2000] reported that multi-reducer sort operations in the Sort library fail unless
the input PCollection is materialized prior to executing the sort.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message