hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-19267) Create/Replicate ACID Write event
Date Sun, 22 Apr 2018 17:24:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-19267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16447297#comment-16447297
] 

ASF GitHub Bot commented on HIVE-19267:
---------------------------------------

GitHub user maheshk114 opened a pull request:

    https://github.com/apache/hive/pull/335

    HIVE-19267 : Create/Replicate ACID Write event

    Replicate ACID write Events
    Create new EVENT_WRITE event with related message format to log the write operations with
in a txn along with data associated.
    Log this event when perform any writes (insert into, insert overwrite, load table, delete,
update, merge, truncate) on table/partition.
    If a single MERGE/UPDATE/INSERT/DELETE statement operates on multiple partitions, then
need to log one event per partition.
    DbNotificationListener should log this type of event to special metastore table named
"MTxnWriteNotificationLog".
    This table should maintain a map of txn ID against list of tables/partitions written by
given txn.
    The entry for a given txn should be removed by the cleaner thread that removes the expired
events from EventNotificationTable.
    Replicate Commit Txn operation (with writes)
    Add new EVENT_COMMIT_TXN to log the metadata/data of all tables/partitions modified within
the txn.
    
    Source warehouse:
    
    This event should read the EVENT_WRITEs from "MTxnWriteNotificationLog" metastore table
to consolidate the list of tables/partitions modified within this txn scope.
    Based on the list of tables/partitions modified and table Write ID, need to compute the
list of delta files added by this txn.
    Repl dump should read this message and dump the metadata and delta files list.
    Target warehouse:
    
    Ensure snapshot isolation at target for on-going read txns which shouldn't view the data
replicated from committed txn. (Ensured with open and allocate write ID events).

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/maheshk114/hive BUG-92690

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/hive/pull/335.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #335
    
----
commit 4cf06e4cd921028cd02d71ba022385986e0227ee
Author: Mahesh Kumar Behera <mbehera@...>
Date:   2018-04-18T06:07:38Z

    HIVE-19267 : Create/Replicate ACID Write event

----


> Create/Replicate ACID Write event
> ---------------------------------
>
>                 Key: HIVE-19267
>                 URL: https://issues.apache.org/jira/browse/HIVE-19267
>             Project: Hive
>          Issue Type: Sub-task
>          Components: repl, Transactions
>    Affects Versions: 3.0.0
>            Reporter: mahesh kumar behera
>            Assignee: mahesh kumar behera
>            Priority: Major
>              Labels: ACID, DR, pull-request-available, replication
>             Fix For: 3.0.0
>
>         Attachments: HIVE-19267.01.patch
>
>
>  
> h1. Replicate ACID write Events
>  * Create new EVENT_WRITE event with related message format to log the write operations
with in a txn along with data associated.
>  * Log this event when perform any writes (insert into, insert overwrite, load table,
delete, update, merge, truncate) on table/partition.
>  * If a single MERGE/UPDATE/INSERT/DELETE statement operates on multiple partitions,
then need to log one event per partition.
>  * DbNotificationListener should log this type of event to special metastore table named
"MTxnWriteNotificationLog".
>  * This table should maintain a map of txn ID against list of tables/partitions written
by given txn.
>  * The entry for a given txn should be removed by the cleaner thread that removes the
expired events from EventNotificationTable.
> h1. Replicate Commit Txn operation (with writes)
> Add new EVENT_COMMIT_TXN to log the metadata/data of all tables/partitions modified within
the txn.
> *Source warehouse:*
>  * This event should read the EVENT_WRITEs from "MTxnWriteNotificationLog" metastore
table to consolidate the list of tables/partitions modified within this txn scope.
>  * Based on the list of tables/partitions modified and table Write ID, need to compute
the list of delta files added by this txn.
>  * Repl dump should read this message and dump the metadata and delta files list.
> *Target warehouse:*
>  * Ensure snapshot isolation at target for on-going read txns which shouldn't view the
data replicated from committed txn. (Ensured with open and allocate write ID events).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message