hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rajesh Balamohan <rajesh.balamo...@gmail.com>
Subject Re: Review Request 48159: HIVE-13901: Hivemetastore add partitions can be slow depending on filesystems
Date Fri, 08 Jul 2016 03:52:17 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48159/
-----------------------------------------------------------

(Updated July 8, 2016, 3:52 a.m.)


Review request for hive and Ashutosh Chauhan.


Changes
-------

The earlier patch caused test failures in TestHiveMetastore where it was trying to have duplicate
partitions.  In earlier patch, any entry in addedPartitions was getting nuked in the finally
block, so even valid partitions were getting deleted.

Changes in latest patch:
1. Making use of Map<PartValEqWrapperLite, Boolean> for addedPartitions (in synchronized
hashmap) to take care of the condition mentioned earlier.
2. Still adding entry to addPartitions isn't threadsafe, when something goes wrong between
time of creating the directory and adding it to addedPartitions (e.g killing server). But
that is something which was already existing and patch isn't trying to change the behaviour.

Ran the following for verification with latest patch.

mvn clean install -DskipTests && cd itests && mvn clean install -DskipTests
&& cd hive-unit && mvn test -Dtest=TestEmbeddedHiveMetaStore,TestRemoteHiveMetaStore,TestSetUGIOnBothClientServer,TestSetUGIOnOnlyClient,TestSetUGIOnOnlyServer


Bugs: HIVE-13901
    https://issues.apache.org/jira/browse/HIVE-13901


Repository: hive-git


Description
-------

Depending on FS, creating external tables & adding partitions can be expensive (e.g msck
which adds all partitions).


Diffs (updated)
-----

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b13fc65 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java f45b90d 

Diff: https://reviews.apache.org/r/48159/diff/


Testing
-------


Thanks,

Rajesh Balamohan


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message