From issues-return-160562-archive-asf-public=cust-asf.ponee.io@hive.apache.org Tue Jun 18 08:20:01 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 9ECCC18066B for ; Tue, 18 Jun 2019 10:20:01 +0200 (CEST) Received: (qmail 30826 invoked by uid 500); 18 Jun 2019 08:20:01 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 30795 invoked by uid 99); 18 Jun 2019 08:20:01 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jun 2019 08:20:01 +0000 Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 77AD6E002F for ; Tue, 18 Jun 2019 08:20:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 39C4F245A4 for ; Tue, 18 Jun 2019 08:20:00 +0000 (UTC) Date: Tue, 18 Jun 2019 08:20:00 +0000 (UTC) From: "mahesh kumar behera (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HIVE-21764) REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-21764?page=3Dcom.atlassia= n.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21764: --------------------------------------- Description:=20 REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular e= xpression + inclusion/exclusion list. So, in case of rename table event, th= e event will be ignored if old table doesn't match the pattern but the new = table should be bootstrapped. REPL DUMP should have a mechanism to detect s= uch tables and automatically bootstrap with incremental replication.Also, i= f renamed table is excluded from replication policy, then need to drop the = old table at target as well.=C2=A0 There are 4 scenarios that needs to be handled. # Both new name and old name satisfies the table name pattern filter. # No need to do anything. The incremental event for rename should take car= e of the replication. # Both the names does not satisfy the table name pattern filter. # Both the names are not in the scope of the policy and this nothing needs= to be done. # New name satisfies the pattern but the old name does not. # The table will not be present at the target.=20 # Rename event handler for dump should detect this case and add the new ta= ble name to the list of table for bootstrap. # All the events related to the table (new name) should be ignored. # If there is a drop event for the table (with new name), then remove the = table from the list of tables to be bootstrapped. # In case of rename (double rename) # If the new name satisfies the table pattern, then add the new name to th= e list of tables to be bootstrapped and remove the old name from the list o= f tables to be bootstrapped.=20 # If the new name does not satisfies then just removed the table name from= the list of tables to be bootstrapped. # New name does not satisfies the pattern but the old name satisfies. # Change the rename event to a drop event. was: REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular e= xpression + inclusion/exclusion list. So, in case of rename table event, th= e event will be ignored if old table doesn't match the pattern but the new = table should be bootstrapped. REPL DUMP should have a mechanism to detect s= uch tables and automatically bootstrap with incremental replication.Also, i= f renamed table is excluded from replication policy, then need to drop the = old table at target as well.=C2=A0 There are 4 scenarios that needs to be handled. =C2=A0 # Both new name and old name satisfies the table name pattern filter. * No need to do anything. The incremental event for rename should take car= e of the replication. # Both the names does not satisfy the table name pattern filter. * Both the names are not in the scope of the policy and this nothing needs= to be done. # New name satisfies the pattern but the old name does not. * The table will not be present at the target.=20 * Rename event handler for dump should detect this case and add the new ta= ble name to the list of table for bootstrap. * All the events related to the table (new name) should be ignored. * If there is a drop event for the table (with new name), then remove the = table from the list of tables to be bootstrapped. * In case of rename (double rename) * If the new name satisfies the table pattern, then add the new name to th= e list of tables to be bootstrapped and remove the old name from the list o= f tables to be bootstrapped.=20 * If the new name does not satisfies then just removed the table name from= the list of tables to be bootstrapped. # New name does not satisfies the pattern but the old name satisfies. * Change the rename event to a drop event. > REPL DUMP should detect and bootstrap any rename table events where old t= able was excluded but renamed table is included. > -------------------------------------------------------------------------= ------------------------------------------------ > > Key: HIVE-21764 > URL: https://issues.apache.org/jira/browse/HIVE-21764 > Project: Hive > Issue Type: Sub-task > Components: repl > Reporter: Sankar Hariappan > Assignee: mahesh kumar behera > Priority: Major > Labels: DR, Replication > > REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular= expression + inclusion/exclusion list. So, in case of rename table event, = the event will be ignored if old table doesn't match the pattern but the ne= w table should be bootstrapped. REPL DUMP should have a mechanism to detect= such tables and automatically bootstrap with incremental replication.Also,= if renamed table is excluded from replication policy, then need to drop th= e old table at target as well.=C2=A0 > There are 4 scenarios that needs to be handled. > # Both new name and old name satisfies the table name pattern filter. > # No need to do anything. The incremental event for rename should take c= are of the replication. > # Both the names does not satisfy the table name pattern filter. > # Both the names are not in the scope of the policy and this nothing nee= ds to be done. > # New name satisfies the pattern but the old name does not. > # The table will not be present at the target.=20 > # Rename event handler for dump should detect this case and add the new = table name to the list of table for bootstrap. > # All the events related to the table (new name) should be ignored. > # If there is a drop event for the table (with new name), then remove th= e table from the list of tables to be bootstrapped. > # In case of rename (double rename) > # If the new name satisfies the table pattern, then add the new name to = the list of tables to be bootstrapped and remove the old name from the list= of tables to be bootstrapped.=20 > # If the new name does not satisfies then just removed the table name fr= om the list of tables to be bootstrapped. > # New name does not satisfies the pattern but the old name satisfies. > # Change the rename event to a drop event. -- This message was sent by Atlassian JIRA (v7.6.3#76005)