hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chaoyu Tang" <>
Subject Re: Review Request 24284: HIVE-7441:Custom partition scheme gets rewritten with hive scheme upon concatenate
Date Tue, 05 Aug 2014 22:00:39 GMT

This is an automatically generated e-mail. To reply, visit:

(Updated Aug. 5, 2014, 10 p.m.)

Review request for hive.


Thanks, Szehon, for the review. I made the changes based on the feedback:
1. Remove the space after !
2. Fixed the failed tests. I believe the test failures were from the order of the selected
rows. I changed the test to query row count since we only want to ensure no empty result is
returned after concatenate in the test.

Bugs: HIVE-7441

Repository: hive-git


Problem (See HIVE-7441 description):
When Hive tries to concatenate the partition data files with hdfs cache is disabled (fs.hdfs.impl.disable.cache=true),
the concatenated data file will be created under a partition directory with a name in form
of /part=value (e.g. /b=part1).  If the original partition directory is different (e.g /part1),
querying the concatenated partition returns empty result since the partition location metastore
still points to the original partition directory (/part1).

When hdfs cache is disabled, filesystem equals comparison returns false for two fs objects
even from same filesystem, which causes the original partition directory (/part1) NOT to be
assigned and used as the concatenated partition destination directory.

Before HDFS-4321 is fixed, we compare two filesystems using their uri.

Diffs (updated)

  common/src/java/org/apache/hadoop/hive/common/ 8dcd4cf4432b261dff8d549b07704d75f4f1c769

  metastore/src/java/org/apache/hadoop/hive/metastore/ 3126880aa1a1c5ed93c9576e84c980511e86145b

  ql/src/java/org/apache/hadoop/hive/ql/metadata/ a7e50adf02e795d192a29d918e3ac24665461e06

  ql/src/test/queries/clientpositive/alter_merge_3.q PRE-CREATION 
  ql/src/test/results/clientpositive/alter_merge_3.q.out PRE-CREATION 



1. Manual tests verified the solution fixed the reported issue
2. New added qtests passed
3. Submit the patch and run pre-commit tests to see if there is any regression.


Chaoyu Tang

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message