hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alan Gates (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-262) Pig needs an optimizer
Date Thu, 12 Jun 2008 16:00:47 GMT

    [ https://issues.apache.org/jira/browse/PIG-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12604553#action_12604553
] 

Alan Gates commented on PIG-262:
--------------------------------

In response to Shravan's comment:  "In the java doc you say that PlanOptimizer need not be
subclassed but you do so for the LogicalOptimizer. Wasn't just a plan optimizer with LogicalTransformer
sufficient?"  I decided it was a little cleaner to put all the rules in one place, so that
I could instantiate them for testing and the real code in the same way.  Otherwise everytime
I defined the logical optimizer, I'd have to define all the rules.  I should update the javadoc
comments.

In response to the type checker concerns.  I went back and forth on this, and chose to put
them in the optimizer because it actually transforms the tree, while the validators mostly
check the tree.  I could put this in the validators in front of the type checker (that's where
I originally had it).  Since the cast I'm inserting just mimics whatever the load in front
of it does, hopefully it won't get it wrong (ie we don't need to typecheck again).  One way
or another the optimizer will be rearranging the tree, which will mean inserting some casts,
etc.

Also, I realized I neglected to put include tests that actually use this new functionality.
 I'll be adding those before I submit this patch.



> Pig needs an optimizer
> ----------------------
>
>                 Key: PIG-262
>                 URL: https://issues.apache.org/jira/browse/PIG-262
>             Project: Pig
>          Issue Type: Sub-task
>          Components: impl
>            Reporter: Alan Gates
>            Assignee: Alan Gates
>         Attachments: optimizer.patch
>
>
> We need to add an optimizer to pig.  This will enable us to do some traditional optimizations,
such as filter and projection pushing, join order and execution choices, etc.  It will also
enable optimizations specific to map/reduce (such as using the combiner).
> The optimizer will need to operate at various levels, including the logical, physical,
and possibly map/reduce plan levels.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message