hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcin Kurczych" <mar...@fb.com>
Subject Re: Review Request: Support archiving for multiple partitions if the table is partitioned by multiple columns
Date Tue, 09 Aug 2011 01:28:13 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1259/
-----------------------------------------------------------

(Updated 2011-08-09 01:28:13.568217)


Review request for hive, Paul Yang and namit jain.


Changes
-------

Extracted some code into separate file with archiving utils, so it can be used in hooks more
easily


Summary
-------

Allowing archiving at chosen level. When table is partitioned by ds, hr, min it allows archiving
at ds level, hr level and min level. Corresponding syntaxes are:
ALTER TABLE test ARCHIVE PARTITION (ds='2008-04-08');
ALTER TABLE test ARCHIVE PARTITION (ds='2008-04-08', hr='11');
ALTER TABLE test ARCHIVE PARTITION (ds='2008-04-08', hr='11', min='30');

You cannot do much to archived partitions. You can read them. You cannot write to them / overwrite
them. You can drop single archived partitions, but not parts of bigger archives.


Diffs (updated)
-----

  trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1153271 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1153271 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java
1153271 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/DummyPartition.java 1153271 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1153271 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 1153271 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1153271 
  trunk/metastore/src/gen/thrift/gen-py/hive_metastore/constants.py 1153271 
  trunk/metastore/src/gen/thrift/gen-rb/hive_metastore_constants.rb 1153271 
  trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1153271 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1153271 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java PRE-CREATION 
  trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1153271 
  trunk/data/conf/hive-site.xml 1153271 
  trunk/metastore/if/hive_metastore.thrift 1153271 
  trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.h 1153271 
  trunk/metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.cpp 1153271 
  trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Constants.java
1153271 
  trunk/metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php 1153271

  trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1153271 
  trunk/ql/src/test/queries/clientnegative/archive_insert1.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_insert2.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_insert3.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_insert4.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_multi1.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_multi2.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_multi3.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_multi4.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_multi5.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_multi6.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_multi7.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_partspec1.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_partspec2.q PRE-CREATION 
  trunk/ql/src/test/queries/clientnegative/archive_partspec3.q PRE-CREATION 
  trunk/ql/src/test/queries/clientpositive/archive_corrupt.q PRE-CREATION 
  trunk/ql/src/test/queries/clientpositive/archive_multi.q PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive1.q.out 1153271 
  trunk/ql/src/test/results/clientnegative/archive2.q.out 1153271 
  trunk/ql/src/test/results/clientnegative/archive_insert1.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_insert2.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_insert3.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_insert4.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_multi1.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_multi2.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_multi3.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_multi4.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_multi5.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_multi6.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_multi7.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_partspec1.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_partspec2.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientnegative/archive_partspec3.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientpositive/archive_corrupt.q.out PRE-CREATION 
  trunk/ql/src/test/results/clientpositive/archive_multi.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/1259/diff


Testing
-------

Added unittests for normal uses.
Tried recovery during manual tests (had to simulate exceptions in different places in code,
which shouldn't be committed, that's why it had to be manual testing).


Thanks,

Marcin


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message