hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
Date Wed, 26 Jun 2019 13:18:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=267611&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267611
]

ASF GitHub Bot logged work on HIVE-21886:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 26/Jun/19 13:17
            Start Date: 26/Jun/19 13:17
    Worklog Time Spent: 10m 
      Work Description: sankarh commented on pull request #688: HIVE-21886 : REPL - With table
list - Handle rename events during replace policy
URL: https://github.com/apache/hive/pull/688#discussion_r297658930
 
 

 ##########
 File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AlterTableHandler.java
 ##########
 @@ -92,16 +93,40 @@ private Scenario scenarioType(org.apache.hadoop.hive.metastore.api.Table
before,
     }
   }
 
-  // return true, if event needs to be dumped, else return false.
-  private boolean handleForTableLevelReplication(Context withinContext) {
-    String oldName = before.getTableName();
-    String newName = after.getTableName();
+  private boolean isSetForBootstrapByReplaceHandler(Context withinContext, String tblName)
{
+    return (withinContext.oldReplScope != null)
+            && !(ReplUtils.tableIncludedInReplScope(withinContext.oldReplScope, tblName))
+            && (ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName));
+  }
 
-    if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldName)) {
-      // If the table is renamed after being added to the list of tables to be bootstrapped,
then remove it from the
-      // list of tables to be bootstrapped.
-      boolean oldTableIsPresent = withinContext.removeFromListOfTablesForBootstrap(before.getTableName());
+  // Return true, if event needs to be dumped, else return false.
+  private boolean handleForTableLevelReplication(Context withinContext, String tblName) {
+    // For alter, if the table does not satisfy the new policy then ignore the event. In
case of replace
+    // policy, if the table does not satisfy the old policy, then ignore the event. As, if
the table satisfy the new
+    // policy, then the table will be bootstrapped by replace handler anb if the table does
not satisfy the new policy,
+    // then anyways the table should be ignored.
+    if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblName)) {
+      // In case of replace, it will be dropped during load. In normal case just ignore the
alter event.
+      LOG.debug("Table " + tblName + " does not satisfy the policy");
+      return false;
+    } else if ((withinContext.oldReplScope != null)
 
 Review comment:
   I think, it is redundant as the general rule is if oldName is not matching the old policy,
skip the event. I don't think, it is hard to debug without these logs, as we dump the event
messages and also both old and new ReplScope. Having too many conditions is also makes the
code not readable. Pls check if we can keep it simple.
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 267611)
    Time Spent: 5.5h  (was: 5h 20m)

> REPL - With table list - Handle rename events during replace policy
> -------------------------------------------------------------------
>
>                 Key: HIVE-21886
>                 URL: https://issues.apache.org/jira/browse/HIVE-21886
>             Project: Hive
>          Issue Type: Sub-task
>          Components: repl
>            Reporter: mahesh kumar behera
>            Assignee: mahesh kumar behera
>            Priority: Major
>              Labels: DR, Replication, pull-request-available
>         Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch
>
>          Time Spent: 5.5h
>  Remaining Estimate: 0h
>
> If some rename events are found to be dumped and replayed while replace policy is getting
executed, it needs to take care of the policy inclusion in both the policy for each table
name.
>  1. Create a list of tables to be bootstrapped. 
>   2. During handling of alter table, if the alter type is rename 
>       1. If the old table name is present in the list of table to be bootstrapped,
remove it.
>        2. If the new table name, matches the new policy, add it to the list of tables
to be bootstrapped.
>   3. During handling of drop table
>        1. if the table is in the list of tables to be bootstrapped, then remove it
and ignore the event.
>   4. During other event handling 
>        1. if the table is there in the list of tables to be bootstrapped, then ignore
the event.
>  
> Rename handling during replace policy
>  # Old name not matching old policy – The old table will not be there at the target
cluster. The table will not be returned by get-all-table.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>  #### New name not matching new policy
>  ***** Ignore the event, no need to do anything.
>  #### New name matching new policy
>  ***** The table will be returned by get-all-table. Replace policy handler will bootstrap
this table as its matching new policy and not matching old policy.
>  ***** All the future events will be ignored as part of check added by replace policy
handling.
>  ***** All the event with old table name will anyways be ignored as the old name is not
matching the new policy.
>  ### New name matching old policy
>  #### New name not matching new policy
>  ***** As the new name is not matching the new policy, the table need not be replicated.
>  ***** As the old name is not matching the new policy, the rename events will be ignored.
>  ***** So nothing to be done for this scenario.
>  #### New name matching new policy
>  ***** As the new name is matching both old and new policy, replace handler will not
bootstrap the table.
>  ***** Add the table to the list of tables to be bootstrapped.
>  ***** Ignore all the events with new name.
>  ***** If there is a drop event for the table (with new name), then remove the table
from the the list of table to be bootstrapped.
>  ***** In case of rename event (double rename)
>  ****** If the new name satisfies the table pattern, then add the new name to the list
of tables to be bootstrapped and remove the old name from the list of tables to be bootstrapped.
>  ****** If the new name does not satisfies then just removed the table name from the
list of tables to be bootstrapped.
>  ## Old name is matching new policy – As per replace policy handler, which checks based
on old table, the table should be bootstrapped and event should be ignored. But rename handler
should decide based on new name.The old table name will not be returned by get-all-table,
so replace handler will not d anything for the old table.
>  ### New name not matching old policy
>  #### New name not matching new policy
>  ***** As the old table is not there at target and new name is not matching new policy.
Ignore the event.
>  ***** No need to add the table to the list of tables to be bootstrapped.
>  ***** All the subsequent events will be ignored as the new name is not matching the
new policy.
>  #### New name matching new policy
>  ***** As the new name is not matching old policy but matching new policy, the table
will be bootstrapped by replace policy handler. So rename event need not add this table to
list of table to be bootstrapped.
>  ***** All the future events will be ignored by replace policy handler.
>  ***** For rename event (double rename)
>  ****** If there is a rename, the table (with intermittent new name) will not be present
and thus replace handler will not bootstrap the table.
>  ****** So if the new name (the latest one) is matching the new policy, then add it to
the list of table to be bootstrapped.
>  ****** And If the new name (the latest one)  is not matching the new policy, then just
ignore the event as the  intermittent new name would not have added to the list of table
to be bootstrapped.
>  ### New name matching old policy
>  #### New name not matching new policy
>  ***** Dump the event. The table will be dropped by repl load at the target.
>  #### New name matching new policy
>  ***** Replace handler will not bootstrap this table as the new name is matching both
policies.
>  ***** As old name is not matching the old policy, the table will not be there at target.
The rename event should add the new name to the list of table to be bootstrapped.
>  ***** Subsequent events with new table name should be ignored.
>  ***** Drop events should not be ignored as if the table is present during bootstrapped,
then its a new table and thus should be dropped.
>  ***** In case of rename event (double rename)
>  ****** If the new name satisfies the table pattern, then add the new name to the list
of tables to be bootstrapped and remove the old name from the list of tables to be bootstrapped.
>  ****** If the new name does not satisfies then just removed the table name from the
list of tables to be bootstrapped.
>  # Old name is matching old policy – The old table will be there at the target cluster.
The table will not be returned by get-all-table. Repl load should delete the old table as
it is not matching the new policy.
>  ## Old name is not matching new policy
>  ### New name not matching old policy
>  #### New name not matching new policy
>  ***** Nothing to be done. Ignore the event.
>  #### New name matching new policy
>  ***** As the new name is not matching old policy but matching new policy, the table
will be bootstrapped by replace policy handler. So rename event need not add this table to
list of table to be bootstrapped.
>  ***** All the future events will be ignored by replace policy handler.
>  ***** For rename event (double rename)
>  ****** If there is a rename, the table (with intermittent new name) will not be present
and thus replace handler will not bootstrap the table.
>  ****** So if the new name (the latest one) is matching the new policy, then add it to
the list of table to be bootstrapped.
>  ****** And If the new name (the latest one)  is not matching the new policy, then just
ignore the event as the  intermittent new name would not have added to the list of table
to be bootstrapped.
>  ### New name matching old policy
>  #### New name not matching new policy
>  ***** Table with new name will be dropped by repl load
>  ***** Along with other event, ignore the rename event also.
>  #### New name matching new policy
>  ***** As the new name is matching both old and new policy, replace handler will not
bootstrap the table.
>  ***** Add the table to the list of tables to be bootstrapped.
>  ***** Ignore all the events with new name.
>  ***** If there is a drop event for the table (with new name), then remove the table
from the the list of table to be bootstrapped.
>  ***** In case of rename event (double rename)
>  ****** If the new name satisfies the table pattern, then add the new name to the list
of tables to be bootstrapped and remove the old name from the list of tables to be bootstrapped.
>  ****** If the new name does not satisfies then just removed the table name from the
list of tables to be bootstrapped.
>  ## Old name is matching new policy
>  ### New name not matching old policy
>  #### New name not matching new policy
>  ***** The old table needs to be dropped at target. Ignore this event, as the old table
is not matching the new policy, it will be dropped by repl load.
>  #### New name matching new policy
>  ***** Allow the event to dump and replayed at target.
>  ***** Allow further events to be handled as usual.
>  ***** In case of rename event (double rename)
>  ****** If the latest new name is matching the new policy, then keep it as is it. Let
rename event replayed at target.
>  ****** If the latest new name is not matching the new policy, then change the rename
event to drop event.
>  ### New name matching old policy
>  #### New name not matching new policy
>  ##### Nothing to be done.
>  #### New name matching new policy
>  ##### Add the table name to the list of tables to be bootstrapped.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message