hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John Pullokkaran" <>
Subject Re: Review Request 12705: HIVE-4878: With Dynamic partitioning, some queries would scan default partition even if query is not using it.
Date Wed, 24 Jul 2013 20:16:17 GMT

This is an automatically generated e-mail. To reply, visit:

(Updated July 24, 2013, 8:15 p.m.)

Review request for hive and Ashutosh Chauhan.


As suggested, new patch discards default partition regardless of the mode.

Repository: hive-git


With Dynamic partitioning, Hive would scan default partitions in some cases even if query
excludes it. As part of partition pruning, predicate is narrowed down to those pieces that
involve partition columns only. This predicate is then evaluated with partition values to
determine, if scan should include those partitions.
But in some cases (like when comparing "_HIVE_DEFAULT_PARTITION_" to numeric data types) expression
evaluation would fail and would return NULL instead of true/false. In such cases the partition
is added to unknown partitions which is then subsequently scanned.

This fix avoids scanning default partition if all of the following is true:
a) Hive dynamic partition mode is strict (hive.exec.dynamic.partition.mode=strict).
b) partition pruning expression failed to evaluate for a given partition.
c) at the least one of the columns in the partition is default partition.

Diffs (updated)

  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/ 6a4a360 
  ql/src/test/queries/clientpositive/dynamic_partition_skip_default.q PRE-CREATION 
  ql/src/test/results/clientpositive/dynamic_partition_skip_default.q.out PRE-CREATION 



Hive Unit Tests Passed.


John Pullokkaran

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message