hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alan Gates (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-1178) LogicalPlan and Optimizer are too complex and hard to work with
Date Fri, 08 Jan 2010 18:16:54 GMT

    [ https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798112#action_12798112

Alan Gates commented on PIG-1178:

Comments on lp.patch:

Is the OperatorPlan that is returned by Rule.Match distinct from the plan that the Transformer
is operating on?  Will the nodes inside that plan be the same nodes (ie that same java objects)
that are in the original plan passed to Rule.Match?  If the answer is yes and yes, then things
will still work, but rule writers will have to be very careful when working with the plans.
 The reason is that the plan returned by Rule.Match will not be the same plan as returned
by Operator.getPlan() when called on the matched nodes.  And rules should use the original
plan (the one returned by Operator.getPlan()) to navigate, as that is the real plan.  I don't
see a better way to do this, so I'm ok with it.  But we will have to document it very well
in the javadocs for writing rules.

In TestExperimentalRule I think we should add tests for two more patterns:
1. a join with two loads as predecessors
2. a spilt with two filters as successors

Also, since I wrote some of this code, we need to have another committers review it before
I check it in.

> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>                 Key: PIG-1178
>                 URL: https://issues.apache.org/jira/browse/PIG-1178
>             Project: Pig
>          Issue Type: Improvement
>            Reporter: Alan Gates
>         Attachments: lp.patch
> The current implementation of the logical plan and the logical optimizer in Pig has proven
to not be easily extensible. Developer feedback has indicated that adding new rules to the
optimizer is quite burdensome. In addition, the logical plan has been an area of numerous
bugs, many of which have been difficult to fix. Developers also feel that the logical plan
is difficult to understand and maintain. The root cause for these issues is that a number
of design decisions that were made as part of the 0.2 rewrite of the front end have now proven
to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and
rebuild the logical plan with a simpler design that will make it much easier to maintain the
logical plan as well as extend the logical optimizer. 
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message