hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eugene Koifman (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-18825) Define ValidTxnList before starting query optimization
Date Tue, 06 Mar 2018 02:02:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-18825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16387145#comment-16387145
] 

Eugene Koifman commented on HIVE-18825:
---------------------------------------

I think there is a problem here that I should've thought of earlier.
Right now we lock in the snapshot after lock acquisition.  This ordering is important if we
ever want to support lock based concurrency control.

Suppose you have 2 concurrent transactions both running "update T1 set x = x + 1".  If we
acquire the update lock first, then record the snapshot, then 2nd txn to get the lock will
see the result of write of the previous one.  If we lock the snapshot before acquiring the
lock, both transactions may lock in exactly the same snapshot and locking becomes useless
as the 2nd will still read an "old" snapshot.

Could the predicates you want be inserted at compile time, but bound to actual values as some
post processing after (or at the end of) {{Driver.acquireLocks()}} as currently implemented?

> Define ValidTxnList before starting query optimization
> ------------------------------------------------------
>
>                 Key: HIVE-18825
>                 URL: https://issues.apache.org/jira/browse/HIVE-18825
>             Project: Hive
>          Issue Type: Improvement
>          Components: Transactions
>    Affects Versions: 3.0.0
>            Reporter: Jesus Camacho Rodriguez
>            Assignee: Jesus Camacho Rodriguez
>            Priority: Major
>         Attachments: HIVE-18825.01.patch, HIVE-18825.02.patch, HIVE-18825.03.patch, HIVE-18825.04.patch,
HIVE-18825.patch
>
>
> Consider a set of tables used by a materialized view where inserts happened after the
materialization was created. To compute incremental view maintenance, we need to be able
to filter only new rows from those base tables. That can be done by inserting a filter
operator with condition e.g. {{ROW\_\_ID.transactionId < highwatermark and ROW\_\_ID.transactionId
NOT IN(<open txns>)}} on top of the MVs query definition and triggering the rewriting
(which should in turn produce a partial rewriting). However, to do that, we need to have a
value for {{ValidTxnList}} during query compilation so we know the snapshot that we are querying.
> This patch aims to generate {{ValidTxnList}} before query optimization. There should
not be any visible changes for end user.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message