hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saket Saurabh <ssaur...@apache.org>
Subject Re: Review Request 49766: HIVE-14035 Enable predicate pushdown to delta files created by ACID Transactions
Date Tue, 26 Jul 2016 18:30:26 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49766/
-----------------------------------------------------------

(Updated July 26, 2016, 11:30 a.m.)


Review request for hive and Eugene Koifman.


Changes
-------

Add more UTs to specifically test AcidUtils and various other compaction scenarios.


Repository: hive-git


Description
-------

https://issues.apache.org/jira/browse/HIVE-14035

In current Hive version, delta files created by ACID transactions do not allow predicate pushdown
if they contain any update/delete events. This is done to preserve correctness when following
a multi-version approach during event collapsing, where an update event overwrites an existing
insert event. 
This JIRA proposes to split an update event into a combination of a delete event followed
by a new insert event, that can enable predicate push down to all delta files without breaking
correctness. To support backward compatibility for this feature, this JIRA also proposes to
add some sort of versioning to ACID that can allow different versions of ACID transactions
to co-exist together.


Diffs (updated)
-----

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e92466f 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java
14f7316 
  hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/AbstractRecordWriter.java
974c6b8 
  itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
ca2a912 
  metastore/if/hive_metastore.thrift 4d92b73 
  metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.h ae14bd1 
  metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.cpp f982bf2 
  metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/hive_metastoreConstants.java
5a666f2 
  metastore/src/gen/thrift/gen-php/metastore/Types.php f505208 
  metastore/src/gen/thrift/gen-py/hive_metastore/constants.py d1c07a5 
  metastore/src/gen/thrift/gen-rb/hive_metastore_constants.rb eeccc84 
  metastore/src/java/org/apache/hadoop/hive/metastore/TransactionalValidationListener.java
3e74675 
  orc/src/java/org/apache/orc/impl/TreeReaderFactory.java c4a2093 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java db6848a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 57b6c67 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 23a13d6 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java c150ec5 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 945b828 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 63d02fb 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java b0f8c8b 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 1a1af28 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 9d927bd 
  ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 8cf261d 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 6caca98 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java d48e441 
  ql/src/test/org/apache/hadoop/hive/ql/io/TestAcidUtils.java b83cea4 

Diff: https://reviews.apache.org/r/49766/diff/


Testing
-------

Tests for the feature are in ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java.
These are mostly integration tests that test end-to-end insert/update/delete scenarios followed
by compaction and cleaning.


Thanks,

Saket Saurabh


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message