hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prasad Chakka (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HIVE-578) Refactor partition pruning code as an optimizer transformation
Date Thu, 23 Jul 2009 02:17:14 GMT

    [ https://issues.apache.org/jira/browse/HIVE-578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734428#action_12734428
] 

Prasad Chakka commented on HIVE-578:
------------------------------------

Some comments.

bq. for(String partName: Hive.get().getPartitionNames(MetaStoreUtils.DEFAULT_DATABASE_NAME,
tab.getName(), (short) -1))

1. should use tab.getDbName() instead.

2. PartitionPruner::prune() should return partSpecs or partNames instead of partition objects
and avoid calling getPartition() on all partNames. this would lower the unnecessary load on
metastore. 

bq. {code}      if (HiveConf.getBoolVar(hiveConf, HiveConf.ConfVars.HIVEOPTPPD)) {
       transformations.add(new PredicatePushDown());
+      transformations.add(new PartitionPruner());
     }

{code}

3. you should add new config param instead of piggybacking on PPD.

4. isn't ColumnInfo.internalName unique among all tables? is tableName an additional information
that is needed? Can't you get this info from RowResolver? Same question on exprNodeColDesc.

5. don't understand the change in DefaultGraphWalker.java

6. can't most of the functionality from ql/parse/PartitionPruner.java class can't be moved
to the new class. atleast change the name so that it is not confusing as which pruner is doing
what.

7. import org.apache.hadoop.hive.ql.ppd.PredicatePushDown; is not needed in SemanticAnalyzer.java

8. 



> Refactor partition pruning code as an optimizer transformation
> --------------------------------------------------------------
>
>                 Key: HIVE-578
>                 URL: https://issues.apache.org/jira/browse/HIVE-578
>             Project: Hadoop Hive
>          Issue Type: Bug
>          Components: Query Processor
>    Affects Versions: 0.3.0
>            Reporter: Ashish Thusoo
>            Assignee: Ashish Thusoo
>         Attachments: patch-578.txt, patch-578_1.txt
>
>
> Some bugs with partition pruning have been reported and the correct fix for many of them
is to rewrite the partition pruning code as an optimizer transformation which gets kicked
in after the predicate pushdown code. This refactor also uses the graph walker framework so
that the partition pruning code gets consolidated well with the frameworks and does not work
on the query block but rather works on the operator tree.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message