hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Namit Jain (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HIVE-578) Refator partition pruning code as an optimizer transformation in order to fix bugs that appear in the current code
Date Wed, 24 Jun 2009 23:46:07 GMT

    [ https://issues.apache.org/jira/browse/HIVE-578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12723804#action_12723804
] 

Namit Jain commented on HIVE-578:
---------------------------------

Some high-level comments:

1. Can you share some expression walker code from Predicate Pushdown ? seems like a lot of
common stuff.
2. Can you always assume that the predicate to be filtered will always be after the tablescan
? 

     Currently, all optimizations are independent of each other, and can be turned off - this
way, 
     you are dependent on a new feature -- may have stability issues.

     I don't remember exactly how does predicate pushdown work - if a filter is already present
immediately after the table
     scan, will the pushed up filter be merged. Is it pushed across user defined mappers/reducers
? Cant these things change
     in the future ?

     It might be a good idea to look for TS*FIL, and then keep some mapping - or probably
use the same mapping that Predicate
     pushdown has to figure out whether the filter is for the table under consideration.

> Refator partition pruning code as an optimizer transformation in order to fix bugs that
appear in the current code
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-578
>                 URL: https://issues.apache.org/jira/browse/HIVE-578
>             Project: Hadoop Hive
>          Issue Type: Bug
>          Components: Query Processor
>    Affects Versions: 0.3.0
>            Reporter: Ashish Thusoo
>            Assignee: Ashish Thusoo
>         Attachments: patch-578.txt
>
>
> Some bugs with partition pruning have been reported and the correct fix for many of them
is to rewrite the partition pruning code as an optimizer transformation which gets kicked
in after the predicate pushdown code. This refactor also uses the graph walker framework so
that the partition pruning code gets consolidated well with the frameworks and does not work
on the query block but rather works on the operator tree.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message