hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ning Zhang" <nzh...@fb.com>
Subject Re: Review Request: HIVE-2049. Push down partition pruning to JDO filtering for a subset of partition predicates
Date Wed, 16 Mar 2011 23:50:15 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/489/
-----------------------------------------------------------

(Updated 2011-03-16 16:50:15.347839)


Review request for hive.


Changes
-------

Add more changes to suite the current implementation of JDO filtering. Some major changes
are:

  - ExpressionTree.makeFilterForEquals(): previous '=' are translated to JDO method matches().
This will introduce false positives if the partition value contains regex special characters
(e.g., dot). I changed this function to use startWith(), endsWith(), and indexOf() in the
case of whether the partition column is at the beginning, end or middle of the partition spec
string. Two unit tests files (ppr_pushdown*.q) are added to test these cases. 
  - ObjectStore.listMPartitions(): added a query.setOrder() to return partitions ordered by
their partition names. This is to be backward compatible with the old partition pruning behavior.

  - PartitionPruner.prune(): check if the partition pruning expression contains non-partition
columns. If so add the resulting partitions to unkn_parts, otherwise to true_parts. This is
required by down stream optimizations. 
  - Utilities.checkJDOPushDown(): return true only if the partition column type is string
and constant type is string. This is required by the current implementation of JDO filter
(see ExpressionTree.java and Filter.g). 

Passed all unit tests. 


Summary
-------

    *  expose HiveMetaStoreClient.listPartitionsByFilter() to Hive.java so that PartitionPruner
can use that for certain partition predicates.
    * only allows {=, AND, OR} in the partition predicates that can be pushed down to JDO
filtering.


Diffs (updated)
-----

  trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1081948

  trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1081948 
  trunk/metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java 1081948

  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1081948 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1081948 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 1081948 
  trunk/ql/src/test/queries/clientpositive/ppr_pushdown.q PRE-CREATION 
  trunk/ql/src/test/queries/clientpositive/ppr_pushdown2.q PRE-CREATION 
  trunk/ql/src/test/results/clientpositive/ppr_pushdown.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientpositive/ppr_pushdown2.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/489/diff


Testing
-------


Thanks,

Ning


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message