hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Dai (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-1132) Column Pruner issues in dealing with unprunable loader
Date Tue, 08 Dec 2009 20:19:18 GMT

    [ https://issues.apache.org/jira/browse/PIG-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787716#action_12787716

Daniel Dai commented on PIG-1132:

Yes, it could happen for any logical operators after Load. The key issue for this problem
is OperatorPlan.insertBetween assume logical plan in a consistent state. What I want to do
here is inserting a LOForEach after LOLoad if the loader do not support pruning. I will need
to achieve this in two steps:

1. insert a LOForEach after LoLoad
2. prune columns the logical operator below

I have to leave the logical plan in an inconsistent state in the middle. If I insert LOForEach
first, then the underlying logical operator has not been pruned yet; If I prune underlying
logical operator first, then there is a schema gap between underlying logical operator and
LOLoad. Both situations are not allowed for insertBetween. I have to use a trick to get around
this problem.

> Column Pruner issues in dealing with unprunable loader
> ------------------------------------------------------
>                 Key: PIG-1132
>                 URL: https://issues.apache.org/jira/browse/PIG-1132
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.4.0
>            Reporter: Daniel Dai
>            Assignee: Daniel Dai
>             Fix For: 0.6.0
>         Attachments: PIG-1132-1.patch, PIG-1132-2.patch
> The following script fail:
> {code}
> a = load '1.txt' using BinStorage() as (a0, a1, a2);
> b = foreach a generate a2, a0, a1;
> c = foreach b generate a0, a2;
> explain c;
> {code}
> Error message:
> ERROR 2163: Error during fixing projections. Could not locate replacement column for
column: 1 in the new predecessor        at org.apache.pig.impl.logicalLayer.ProjectFixerUpper.visit(ProjectFixerUpper.java:262)
>         at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:404)
>         at org.apache.pig.impl.logicalLayer.LOProject.visit(LOProject.java:58)
>         at org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67)
>         at org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:50)
>         at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
>         at org.apache.pig.impl.logicalLayer.LOForEach.rewire(LOForEach.java:770)
>         ... 19 more
> Also even we fix the exception, the result is wrong.
> Thanks Thejas for finding a reproducible test case for PigLatin

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message