hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eugene Koifman (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-18747) Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.
Date Thu, 29 Mar 2018 23:46:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-18747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419947#comment-16419947
] 

Eugene Koifman commented on HIVE-18747:
---------------------------------------

+1 patch 3

is the change in TxnHandler.getValidWriteIdsForTable() specific to this ticket or a general
fix?

> Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.
> ------------------------------------------------------------------
>
>                 Key: HIVE-18747
>                 URL: https://issues.apache.org/jira/browse/HIVE-18747
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Transactions
>    Affects Versions: 3.0.0
>            Reporter: Sankar Hariappan
>            Assignee: Sankar Hariappan
>            Priority: Minor
>              Labels: ACID, pull-request-available
>             Fix For: 3.0.0
>
>         Attachments: HIVE-18747.01.patch, HIVE-18747.02.patch, HIVE-18747.03.patch
>
>
> Per table write ID implementation (HIVE-18192) maintains a map between txn ID and table
write ID in TXN_TO_WRITE_ID meta table. 
> The entries in this table is used to generate ValidWriteIdList for the given ValidTxnList
to ensure snapshot isolation. 
> When table or database is dropped, then these entries are cleaned-up. But, it is necessary
to clean-up for active tables too for better performance.
> TXN_TO_WRITE_ID table keeps a mapping of Transaction ID to Write ID.  The state of each
Write ID (open, committed, aborted) is determined by the state of the parent transaction.
 In order to be able to get a WriteIdList that is accurate wrt ValidTxnList that is locked
in at the start of the transaction, we have to retain txnid<->writeid mapping even after
the transaction ends. This is because a reader at Snapshot Isolation that started when transaction
X was open, should continue to ignore the data written by X even after X commits.
> So we need a mechanism to know when it is safe to remove TXN_TO_WRITE_ID.  There are
2 parts to it. When txn X is opened, it records Y=select min(txn_id) from TXNS where txn_state=’o’
in MIN_HISTORY(txnid,opentxnid) table, i.e. it adds (X, Y) to MIN_HISTORY.  On commit (and
abort) of X, it removes its own entry from MIN_HISTORY. In the absence of Aborted transactions,
MIN_HISTORY gives us the smallest open txnid across all active reader snapshots.  Let Z=select
min(opentxnid) from MIN_HISTORY. We can delete entries from TXN_TO_WRITE_ID once TXN_TO_WRITE_ID.T2W_TXNID
< Z since every active reader sees txns < Z as committed.
> If S is aborted txns, we retain the metadata about it in TXNS as long as any data written
S may be visible to some reader in the system so that the reader knows to skip this data.
 The rules for when that is are complex but wrt to TXN_TO_WRITE_ID, if A=select min(TXN_ID)
from TXNS where TXN_STATE=’a’, then it’s safe to delete from TXN_TO_WRITE_ID when TXN_TO_WRITE_ID.T2W_TXNID
< min(Z,A).  
> If no open or aborted txns exist in the system, then we need to enable cleanup using
latest allocated value of NEXT_TXN_ID table. Delete condition would be TXN_TO_WRITE_ID.T2W_TXNID
< min(Z,A,NEXT_TXN_ID.ntxn_next).  
> Also, it is proposed to trigger cleanup on TXN_TO_WRITE_ID from initiator immediately
after cleaning up aborted txns metadata from TXNS table.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message