hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Olga Natkovich (JIRA)" <j...@apache.org>
Subject [jira] Updated: (PIG-362) Limit can not push in front of ForEach with flatten
Date Tue, 19 Aug 2008 00:09:44 GMT

     [ https://issues.apache.org/jira/browse/PIG-362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Olga Natkovich updated PIG-362:

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

patch committed. thanks daniel for contributing!

> Limit can not push in front of ForEach with flatten
> ---------------------------------------------------
>                 Key: PIG-362
>                 URL: https://issues.apache.org/jira/browse/PIG-362
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>    Affects Versions: types_branch
>            Reporter: Daniel Dai
>            Assignee: Daniel Dai
>             Fix For: types_branch
>         Attachments: PIG-362-2.patch, PIG-362.patch
> Currently logical optimizer will push Limit in front of ForEach with flatten. It is based
on the assumption that ForEach with a flatten always increase the number of records. However,
this is a false assumption. In the case that there is empty bags inside input tuple, the number
of output records can be 0, thus less than input records. 
> We have no way to know whether there is an empty bag in the input at optimization time.
So the only solution is not to push Limit in front of ForEach with flatten

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message