hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saket Saurabh <ssaur...@apache.org>
Subject Re: Review Request 49766: HIVE-14035 Enable predicate pushdown to delta files created by ACID Transactions
Date Tue, 09 Aug 2016 01:10:07 GMT


> On Aug. 4, 2016, 2:29 p.m., Sergey Shelukhin wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java, line 399
> > <https://reviews.apache.org/r/49766/diff/4/?file=1455576#file1455576line399>
> >
> >     rowId is always -1 here. Intended?

@Sergey: Thanks for pointing this out. It was a bug and not intended.


- Saket


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49766/#review144816
-----------------------------------------------------------


On Aug. 8, 2016, 5:53 p.m., Saket Saurabh wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49766/
> -----------------------------------------------------------
> 
> (Updated Aug. 8, 2016, 5:53 p.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> https://issues.apache.org/jira/browse/HIVE-14035
> 
> In current Hive version, delta files created by ACID transactions do not allow predicate
pushdown if they contain any update/delete events. This is done to preserve correctness when
following a multi-version approach during event collapsing, where an update event overwrites
an existing insert event. 
> This JIRA proposes to split an update event into a combination of a delete event followed
by a new insert event, that can enable predicate push down to all delta files without breaking
correctness. To support backward compatibility for this feature, this JIRA also proposes to
add some sort of versioning to ACID that can allow different versions of ACID transactions
to co-exist together.
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 70816bd 
>   hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java
14f7316 
>   hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/AbstractRecordWriter.java
974c6b8 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
731caa8 
>   metastore/if/hive_metastore.thrift a2e35b8 
>   metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.h ae14bd1 
>   metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.cpp f982bf2 
>   metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/hive_metastoreConstants.java
5a666f2 
>   metastore/src/gen/thrift/gen-php/metastore/Types.php d6f7f49 
>   metastore/src/gen/thrift/gen-py/hive_metastore/constants.py d1c07a5 
>   metastore/src/gen/thrift/gen-rb/hive_metastore_constants.rb eeccc84 
>   metastore/src/java/org/apache/hadoop/hive/metastore/TransactionalValidationListener.java
3e74675 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java db6848a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 57b6c67 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 26e6443 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidOutputFormat.java dd90a95 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 449d889 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 945b828 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 334cb31 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java efde2db 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 1a1af28 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 8cb5e8a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 8cf261d 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 6caca98 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java af192fb 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands3.java PRE-CREATION 
>   ql/src/test/org/apache/hadoop/hive/ql/io/TestAcidUtils.java 556df18 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 6648829 
> 
> Diff: https://reviews.apache.org/r/49766/diff/
> 
> 
> Testing
> -------
> 
> Tests for the feature are in ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java.
These are mostly integration tests that test end-to-end insert/update/delete scenarios followed
by compaction and cleaning.
> 
> 
> Thanks,
> 
> Saket Saurabh
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message