hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Dai (JIRA)" <j...@apache.org>
Subject [jira] Updated: (PIG-262) Pig needs an optimizer
Date Wed, 09 Jul 2008 06:02:33 GMT

     [ https://issues.apache.org/jira/browse/PIG-262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Daniel Dai updated PIG-262:
---------------------------

    Attachment: optimizer2.patch

It is true we should be careful when doing transformation, but I still belive we need it because
many optimizations depend on that. Let's define it more clearly to alleviate the non-deterministic
problem:
1. All matches are taken from the snapshot before any transformation
2. We use well-defined tree walker algorithm: depth-first, dependency-order, etc
3. For every node visited, check all possible matches start from the node

I provide an implementation optimizer2.patch for reference. This is an incremental patch based
on optimizer.patch.

> Pig needs an optimizer
> ----------------------
>
>                 Key: PIG-262
>                 URL: https://issues.apache.org/jira/browse/PIG-262
>             Project: Pig
>          Issue Type: Sub-task
>          Components: impl
>            Reporter: Alan Gates
>            Assignee: Alan Gates
>         Attachments: optimizer.patch, optimizer2.patch
>
>
> We need to add an optimizer to pig.  This will enable us to do some traditional optimizations,
such as filter and projection pushing, join order and execution choices, etc.  It will also
enable optimizations specific to map/reduce (such as using the combiner).
> The optimizer will need to operate at various levels, including the logical, physical,
and possibly map/reduce plan levels.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message