crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Wills (JIRA)" <>
Subject [jira] [Updated] (CRUNCH-241) Write side outputs from the Mapper stage of a MapReduce job
Date Fri, 19 Jul 2013 06:10:48 GMT


Josh Wills updated CRUNCH-241:

    Attachment: CRUNCH-241.patch

The patch. Learned this trick from working with [~srowen].
> Write side outputs from the Mapper stage of a MapReduce job
> -----------------------------------------------------------
>                 Key: CRUNCH-241
>                 URL:
>             Project: Crunch
>          Issue Type: Bug
>          Components: Core
>            Reporter: Josh Wills
>            Assignee: Josh Wills
>         Attachments: CRUNCH-241.patch
> Right now, Crunch always writes output files from the "last" stage of whatever kind of
job it runs: either the reduce-side of a MapReduce job, or the map-side of a map-only job.
This often leads to situations where we have to re-process the same input twice, once for
the map-side outputs and again for the reduce-side outputs.
> This change adds the ability for Crunch to write side outputs from the mapper phase of
a MapReduce job (i.e., we can write output Targets from both the map side and the reduce side.)
This should help lots of pipelines that implement these types of writes execute much faster.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message