hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alan Gates (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-158) Rework logical plan
Date Mon, 07 Apr 2008 22:53:25 GMT

    [ https://issues.apache.org/jira/browse/PIG-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12586582#action_12586582
] 

Alan Gates commented on PIG-158:
--------------------------------

In answer to Shravan's question, consider a script like the following:

{code}
a = load 'myfile';
b = group a by $0;
c = foreach b { cc = distinct $1; generate myudf($1); }
store c into 'output';
{code}

And consider a visitor whose job it is to find all of the UDFs used in a script (for whatever
reason).  If we go with the nested plan model, but we don't allow visitors to switch plans,
then when the outer most visitor (V) comes to the foreach, it has to spawn a child visitor
(VC) to visit the nested plan.  When VC is done, V has to then ask VC what it learned visiting
the nested plan.  (That is it has to incorporate VCs state into its state.)  On the other
hand, if V can instead change walkers when it hits the foreach, it can still visit the nested
plan.  The state change that resulted from visiting the nested plan (in this case, learning
that the UDF myudf is called) need not be transmitted between visitors.


> Rework logical plan
> -------------------
>
>                 Key: PIG-158
>                 URL: https://issues.apache.org/jira/browse/PIG-158
>             Project: Pig
>          Issue Type: Sub-task
>          Components: impl
>            Reporter: Alan Gates
>            Assignee: Alan Gates
>         Attachments: logical_operators.patch, logical_operators_rev_1.patch, logical_operators_rev_2.patch,
logical_operators_rev_3.patch, visitorWalker.patch
>
>
> Rework the logical plan in line with http://wiki.apache.org/pig/PigExecutionModel

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message