hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Navis Ryu" <navis....@nexr.com>
Subject Re: Review Request 19903: Support bulk deleting directories for partition drop with partial spec
Date Thu, 03 Apr 2014 02:09:39 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19903/
-----------------------------------------------------------

(Updated April 3, 2014, 2:09 a.m.)


Review request for hive.


Changes
-------

Addressed comments & Fixed test fails


Bugs: HIVE-6809
    https://issues.apache.org/jira/browse/HIVE-6809


Repository: hive-git


Description
-------

In busy hadoop system, dropping many of partitions takes much more time than expected. In
hive-0.11.0, removing 1700 partitions by single partial spec took 90 minutes, which is reduced
to 3 minutes when deleteData is set false. I couldn't test this in recent hive, which has
HIVE-6256 but if the time-taking part is mostly from removing directories, it seemed not helpful
to reduce whole processing time.


Diffs (updated)
-----

  metastore/if/hive_metastore.thrift eef1b80 
  metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h 2a1b4d7 
  metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp 9567874 
  metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp b18009c 
  metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
4f051af 
  metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php c79624f 
  metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote fdedb57 
  metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py 23679be 
  metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb 56c23e6 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 27077b4 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 664dccd 
  metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 0c2209b 
  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 6a0eabe 
  metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java e0de0e0 
  metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java 5c00aa1

  metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 5025b83

  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 5cb030c 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 5d5fa78 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java a73a5e0 
  ql/src/java/org/apache/hadoop/hive/ql/plan/DropTableDesc.java ba30e1f 
  ql/src/test/queries/clientpositive/drop_partitions_partialspec.q PRE-CREATION 
  ql/src/test/results/clientnegative/drop_partition_failure.q.out b94c6c2 
  ql/src/test/results/clientnegative/drop_partition_filter_failure.q.out 0ab5e02 
  ql/src/test/results/clientpositive/drop_multi_partitions.q.out 735920b 
  ql/src/test/results/clientpositive/drop_partitions_partialspec.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/19903/diff/


Testing
-------


Thanks,

Navis Ryu


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message