hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alan Gates (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-158) Rework logical plan
Date Wed, 02 Apr 2008 19:53:24 GMT

    [ https://issues.apache.org/jira/browse/PIG-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12584744#action_12584744
] 

Alan Gates commented on PIG-158:
--------------------------------

I like Pi's idea of operator containing plans of operators in that it does fit our nested
model well.  My concern is state in our visitors.  Right now a visitor contains a reference
to the plan it is visiting because it must traverse the plan.  When the visitor comes to a
node that contains a plan, how does it then visit that plan?  I don't want to have to spawn
a new visitor, because if there's information learned or state changed in that second visitor,
it has to somehow be transmitted back to the original visitor.

One possibility is to follow another suggestion that Pi gave, which is to separate walkers
and visitors.  A walker would be associated with the plan.  A visitor would keep a stack of
walkers.  When it encountered a nested plan, it would push its current walker on the stack,
spawn a new walker with the nested plan, and visit that nested plan.  When it finished it
could pop the previous walker back off the stack and continue. 

In this proposal visiting a filter operator would look something like this:

{code}
class LOVisitor {
    visit(Filter f) {
        Walker w = new Walker(f.getComparisonPlan());
        walkers.push(currentWalker);
        currentWalker = w;
        f.getComparisonPlan().getRoot().visit(this);
        currentWalker = walkers.pop();
    }
}

{code}

This allows us to have nested plans, keep state across visitations of nested plans, and not
go through separating expression operators from logical operators.

On the down side, it does make visitor logic a more complex.  Anyone see any other downsides?
        


> Rework logical plan
> -------------------
>
>                 Key: PIG-158
>                 URL: https://issues.apache.org/jira/browse/PIG-158
>             Project: Pig
>          Issue Type: Sub-task
>          Components: impl
>            Reporter: Alan Gates
>            Assignee: Alan Gates
>         Attachments: logical_operators.patch, logical_operators_rev_1.patch, logical_operators_rev_2.patch,
logical_operators_rev_3.patch
>
>
> Rework the logical plan in line with http://wiki.apache.org/pig/PigExecutionModel

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message