hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rajesh Balamohan <>
Subject Re: Review Request 48159: HIVE-13901: Hivemetastore add partitions can be slow depending on filesystems
Date Fri, 08 Jul 2016 03:52:17 GMT

This is an automatically generated e-mail. To reply, visit:

(Updated July 8, 2016, 3:52 a.m.)

Review request for hive and Ashutosh Chauhan.


The earlier patch caused test failures in TestHiveMetastore where it was trying to have duplicate
partitions.  In earlier patch, any entry in addedPartitions was getting nuked in the finally
block, so even valid partitions were getting deleted.

Changes in latest patch:
1. Making use of Map<PartValEqWrapperLite, Boolean> for addedPartitions (in synchronized
hashmap) to take care of the condition mentioned earlier.
2. Still adding entry to addPartitions isn't threadsafe, when something goes wrong between
time of creating the directory and adding it to addedPartitions (e.g killing server). But
that is something which was already existing and patch isn't trying to change the behaviour.

Ran the following for verification with latest patch.

mvn clean install -DskipTests && cd itests && mvn clean install -DskipTests
&& cd hive-unit && mvn test -Dtest=TestEmbeddedHiveMetaStore,TestRemoteHiveMetaStore,TestSetUGIOnBothClientServer,TestSetUGIOnOnlyClient,TestSetUGIOnOnlyServer

Bugs: HIVE-13901

Repository: hive-git


Depending on FS, creating external tables & adding partitions can be expensive (e.g msck
which adds all partitions).

Diffs (updated)

  common/src/java/org/apache/hadoop/hive/conf/ b13fc65 
  metastore/src/java/org/apache/hadoop/hive/metastore/ f45b90d 




Rajesh Balamohan

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message