Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Content-Type: multipart/alternative;
 boundary="===============8488491042886663555=="
MIME-Version: 1.0
Subject: Re: Review Request 49766: HIVE-14035 Enable predicate pushdown to
 delta files created by ACID Transactions
From: Saket Saurabh <ssaurabh@apache.org>
To: Eugene Koifman <ekoifman@hortonworks.com>
Cc: Saket Saurabh <ssaurabh@apache.org>, hive <dev@hive.apache.org>,
 Lefty Leverenz <leftyleverenz@gmail.com>
Date: Wed, 27 Jul 2016 21:58:29 -0000
Message-ID: <20160727215829.11791.43773@reviews.apache.org>
Auto-Submitted: auto-generated
Sender: Saket Saurabh <noreply@reviews.apache.org>
References: <20160721074225.15322.29845@reviews.apache.org>
In-Reply-To: <20160721074225.15322.29845@reviews.apache.org>
Reply-To: Saket Saurabh <ssaurabh@apache.org>
archived-at: Wed, 27 Jul 2016 21:58:38 -0000

--===============8488491042886663555==
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit


> On July 21, 2016, 12:42 a.m., Lefty Leverenz wrote:
> >

@Lefty, does the updated description for this config variable seem to explain better now?


- Saket


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49766/#review143059
-----------------------------------------------------------


On July 27, 2016, 2:54 p.m., Saket Saurabh wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49766/
> -----------------------------------------------------------
> 
> (Updated July 27, 2016, 2:54 p.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> https://issues.apache.org/jira/browse/HIVE-14035
> 
> In current Hive version, delta files created by ACID transactions do not allow predicate pushdown if they contain any update/delete events. This is done to preserve correctness when following a multi-version approach during event collapsing, where an update event overwrites an existing insert event. 
> This JIRA proposes to split an update event into a combination of a delete event followed by a new insert event, that can enable predicate push down to all delta files without breaking correctness. To support backward compatibility for this feature, this JIRA also proposes to add some sort of versioning to ACID that can allow different versions of ACID transactions to co-exist together.
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e92466f 
>   hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java 14f7316 
>   hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/AbstractRecordWriter.java 974c6b8 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java ca2a912 
>   metastore/if/hive_metastore.thrift 4d92b73 
>   metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.h ae14bd1 
>   metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.cpp f982bf2 
>   metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/hive_metastoreConstants.java 5a666f2 
>   metastore/src/gen/thrift/gen-php/metastore/Types.php f505208 
>   metastore/src/gen/thrift/gen-py/hive_metastore/constants.py d1c07a5 
>   metastore/src/gen/thrift/gen-rb/hive_metastore_constants.rb eeccc84 
>   metastore/src/java/org/apache/hadoop/hive/metastore/TransactionalValidationListener.java 3e74675 
>   orc/src/java/org/apache/orc/impl/TreeReaderFactory.java c4a2093 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java db6848a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 57b6c67 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 23a13d6 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidOutputFormat.java dd90a95 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java c150ec5 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 945b828 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 63d02fb 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 1a1af28 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 9d927bd 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 8cf261d 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 6caca98 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java d48e441 
>   ql/src/test/org/apache/hadoop/hive/ql/io/TestAcidUtils.java b83cea4 
> 
> Diff: https://reviews.apache.org/r/49766/diff/
> 
> 
> Testing
> -------
> 
> Tests for the feature are in ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java. These are mostly integration tests that test end-to-end insert/update/delete scenarios followed by compaction and cleaning.
> 
> 
> Thanks,
> 
> Saket Saurabh
> 
>


--===============8488491042886663555==--