hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alan Gates (JIRA)" <j...@apache.org>
Subject [jira] Updated: (PIG-304) Distinct fail if previous map plan is closed
Date Mon, 14 Jul 2008 17:14:33 GMT

     [ https://issues.apache.org/jira/browse/PIG-304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Alan Gates updated PIG-304:

    Attachment: movelocalrearrange.patch

This patch adds a post MR compile visitor that moves local rearranges from the reducer to
the next mapper.  This is not the best solution.  The best solution would be to make the maps
somehow no-ops, so that we can avoid that stage.  Ways to do that need further investigation.
 The next best solution would be to correct the logic in MRCompiler to place the local rearranges
in the subsequent mapper instead of the reducer.  But Shravan is out until Aug 4 and I don't
want to change that code without his input.  And seems to work.

> Distinct fail if previous map plan is closed
> --------------------------------------------
>                 Key: PIG-304
>                 URL: https://issues.apache.org/jira/browse/PIG-304
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>    Affects Versions: types_branch
>            Reporter: Daniel Dai
>             Fix For: types_branch
>         Attachments: distinct.patch, movelocalrearrange.patch
> Distinct will fail when the previous map plan is closed. For example, the following script
> a = load 'a';
> b = group a by $0;
> c = foreach b generate $1;
> d = distinct c;
> dump d;

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message