hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashutosh Chauhan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-747) Logical to Physical Plan Translation fails when temporary alias are created within foreach
Date Sat, 05 Dec 2009 19:24:20 GMT

    [ https://issues.apache.org/jira/browse/PIG-747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12786466#action_12786466
] 

Ashutosh Chauhan commented on PIG-747:
--------------------------------------

In my opinion option 2 is better then option 1. Instead of carrying references, copies should
be made as early as possible, right at time of parsing. Among other things this impacts the
cloning of operators. And if there are references, references gets cloned which is not ideal
situation. If there are different operators with different operator keys, one need not to
worry about dangling references getting cloned. 
Secondly, if plan is modified by optimizer by moving around operators in a tree, it so much
easier to work with copy of operators instead of references to same object. 
In terms of implementation, I have no idea which of the two will be easier to do.

> Logical to Physical Plan Translation fails when temporary alias are created within foreach
> ------------------------------------------------------------------------------------------
>
>                 Key: PIG-747
>                 URL: https://issues.apache.org/jira/browse/PIG-747
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.4.0
>            Reporter: Viraj Bhat
>            Assignee: Daniel Dai
>             Fix For: 0.7.0
>
>         Attachments: physicalplan.txt, physicalplanprob.pig, PIG-747-1.patch
>
>
> Consider a the pig script which calculates a new column F inside the foreach as:
> {code}
> A = load 'physicalplan.txt' as (col1,col2,col3);
> B = foreach A {
>    D = col1/col2;
>    E = col3/col2;
>    F = E - (D*D);
>    generate
>    F as newcol;
> };
> dump B;
> {code}
> This gives the following error:
> =======================================================================================================================================
> Caused by: org.apache.pig.backend.hadoop.executionengine.physicalLayer.LogicalToPhysicalTranslatorException:
ERROR 2015: Invalid physical operators in the physical plan
>         at org.apache.pig.backend.hadoop.executionengine.physicalLayer.LogToPhyTranslationVisitor.visit(LogToPhyTranslationVisitor.java:377)
>         at org.apache.pig.impl.logicalLayer.LOMultiply.visit(LOMultiply.java:63)
>         at org.apache.pig.impl.logicalLayer.LOMultiply.visit(LOMultiply.java:29)
>         at org.apache.pig.impl.plan.DependencyOrderWalkerWOSeenChk.walk(DependencyOrderWalkerWOSeenChk.java:68)
>         at org.apache.pig.backend.hadoop.executionengine.physicalLayer.LogToPhyTranslationVisitor.visit(LogToPhyTranslationVisitor.java:908)
>         at org.apache.pig.impl.logicalLayer.LOForEach.visit(LOForEach.java:122)
>         at org.apache.pig.impl.logicalLayer.LOForEach.visit(LOForEach.java:41)
>         at org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:68)
>         at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
>         at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:246)
>         ... 10 more
> Caused by: org.apache.pig.impl.plan.PlanException: ERROR 0: Attempt to give operator
of type org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.Divide
multiple outputs.  This operator does not support multiple outputs.
>         at org.apache.pig.impl.plan.OperatorPlan.connect(OperatorPlan.java:158)
>         at org.apache.pig.backend.hadoop.executionengine.physicalLayer.plans.PhysicalPlan.connect(PhysicalPlan.java:89)
>         at org.apache.pig.backend.hadoop.executionengine.physicalLayer.LogToPhyTranslationVisitor.visit(LogToPhyTranslationVisitor.java:373)
>         ... 19 more
> =======================================================================================================================================

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message