From issues-return-161419-archive-asf-public=cust-asf.ponee.io@hive.apache.org Wed Jun 26 11:18:03 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 6514E18072F for ; Wed, 26 Jun 2019 13:18:03 +0200 (CEST) Received: (qmail 2840 invoked by uid 500); 26 Jun 2019 11:18:02 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 2831 invoked by uid 99); 26 Jun 2019 11:18:02 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Jun 2019 11:18:02 +0000 Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id F2CAFE2E2F for ; Wed, 26 Jun 2019 11:18:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 65EE425819 for ; Wed, 26 Jun 2019 11:18:01 +0000 (UTC) Date: Wed, 26 Jun 2019 11:18:01 +0000 (UTC) From: "ASF GitHub Bot (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Work logged] (HIVE-21886) REPL - With table list - Handle rename events during replace policy MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-21886?focusedWorklogId=3D= 267494&page=3Dcom.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpa= nel#worklog-267494 ] ASF GitHub Bot logged work on HIVE-21886: ----------------------------------------- Author: ASF GitHub Bot Created on: 26/Jun/19 11:17 Start Date: 26/Jun/19 11:17 Worklog Time Spent: 10m=20 Work Description: sankarh commented on pull request #688: HIVE-21886 = : REPL - With table list - Handle rename events during replace policy URL: https://github.com/apache/hive/pull/688#discussion_r297586465 =20 =20 ########## File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/Al= terTableHandler.java ########## @@ -92,16 +93,40 @@ private Scenario scenarioType(org.apache.hadoop.hive.m= etastore.api.Table before, } } =20 - // return true, if event needs to be dumped, else return false. - private boolean handleForTableLevelReplication(Context withinContext) { - String oldName =3D before.getTableName(); - String newName =3D after.getTableName(); + private boolean isSetForBootstrapByReplaceHandler(Context withinContext,= String tblName) { + return (withinContext.oldReplScope !=3D null) + && !(ReplUtils.tableIncludedInReplScope(withinContext.oldReplS= cope, tblName)) + && (ReplUtils.tableIncludedInReplScope(withinContext.replScope= , tblName)); + } =20 - if (ReplUtils.tableIncludedInReplScope(withinContext.replScope, oldNam= e)) { - // If the table is renamed after being added to the list of tables t= o be bootstrapped, then remove it from the - // list of tables to be bootstrapped. - boolean oldTableIsPresent =3D withinContext.removeFromListOfTablesFo= rBootstrap(before.getTableName()); + // Return true, if event needs to be dumped, else return false. + private boolean handleForTableLevelReplication(Context withinContext, St= ring tblName) { + // For alter, if the table does not satisfy the new policy then ignore= the event. In case of replace + // policy, if the table does not satisfy the old policy, then ignore t= he event. As, if the table satisfy the new + // policy, then the table will be bootstrapped by replace handler anb = if the table does not satisfy the new policy, + // then anyways the table should be ignored. + if (!ReplUtils.tableIncludedInReplScope(withinContext.replScope, tblNa= me)) { + // In case of replace, it will be dropped during load. In normal cas= e just ignore the alter event. + LOG.debug("Table " + tblName + " does not satisfy the policy"); + return false; + } else if ((withinContext.oldReplScope !=3D null) + && (!ReplUtils.tableIncludedInReplScope(withinContext.oldReplS= cope, tblName))) { + LOG.debug("Table " + tblName + " is set for bootstrap"); + return false; + } else { + // Table satisfies both old (if its there) and current policy, dump = the alter event. + return true; + } + } =20 + // Return true, if event needs to be dumped, else return false. + private boolean handleForTableLevelReplicationForRename(Context withinCo= ntext, String oldName, String newName) { =20 Review comment: The method name can be renameHandlerForTableLevelReplication(). =20 ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. =20 For queries about this service, please contact Infrastructure at: users@infra.apache.org Issue Time Tracking ------------------- Worklog Id: (was: 267494) Time Spent: 0.5h (was: 20m) > REPL - With table list - Handle rename events during replace policy > ------------------------------------------------------------------- > > Key: HIVE-21886 > URL: https://issues.apache.org/jira/browse/HIVE-21886 > Project: Hive > Issue Type: Sub-task > Components: repl > Reporter: mahesh kumar behera > Assignee: mahesh kumar behera > Priority: Major > Labels: DR, Replication, pull-request-available > Attachments: HIVE-21886.01.patch, HIVE-21886.02.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > If some rename events are found to be dumped and replayed while replace p= olicy is getting executed, it needs to take care of the policy inclusion in= both the policy for each table name. > =C2=A01. Create a list of tables to be bootstrapped.=C2=A0 > =C2=A0 2. During handling of alter table, if the alter type is rename=C2= =A0 > =C2=A0 =C2=A0 =C2=A0 1. If the old table name is present in the list of t= able to be bootstrapped, remove it. > =C2=A0 =C2=A0 =C2=A0 =C2=A02. If the new table name, matches the new poli= cy, add it to the list of tables to be bootstrapped. > =C2=A0 3. During handling of drop table > =C2=A0 =C2=A0 =C2=A0 =C2=A01. if the table is in the list of tables to be= bootstrapped, then remove it and ignore the event. > =C2=A0 4. During other event handling=C2=A0 > =C2=A0 =C2=A0 =C2=A0 =C2=A01. if the table is there in the list of tables= to be bootstrapped, then ignore the event. > =C2=A0 > Rename handling during replace policy > # Old name not matching old policy =E2=80=93 The old table will not be t= here at the target cluster. The table will not be returned by get-all-table= . > ## Old name is not matching new policy > ### New name not matching old policy > #### New name not matching new policy > ***** Ignore the event, no need to do anything. > #### New name matching new policy > ***** The table will be returned by get-all-table. Replace policy handle= r will bootstrap this table as its matching new policy and not matching old= policy. > ***** All the future events will be ignored as part of check added by re= place policy handling. > ***** All the event with old table name will anyways be ignored as the o= ld name is not matching the new policy. > ### New name matching old policy > #### New name not matching new policy > ***** As the new name is not matching the new policy, the table need not= be replicated. > ***** As the old name is not matching the new policy, the rename events = will be ignored. > ***** So nothing to be done for this scenario. > #### New name matching new policy > ***** As the new name is matching both old and new policy, replace handl= er will not bootstrap the table. > ***** Add the table to the list of tables to be bootstrapped. > ***** Ignore all the events with new name. > ***** If there is a drop event for the table (with new name), then remov= e the table from the the list of table to be bootstrapped. > ***** In case of rename event (double rename) > ****** If the new name satisfies the table pattern, then add the new nam= e to the list of tables to be bootstrapped and remove the old name from the= list of tables to be bootstrapped. > ****** If the new name does not satisfies then just removed the table na= me from the list of tables to be bootstrapped. > ## Old name is matching new policy =E2=80=93 As per replace policy handl= er, which checks based on old table, the table should be bootstrapped and e= vent should be ignored. But rename handler should decide based on new name.= The old table name will not be returned by get-all-table, so replace handle= r will not d anything for the old table. > ### New name not matching old policy > #### New name not matching new policy > ***** As the old table is not there at target and new name is not matchi= ng new policy. Ignore the event. > ***** No need to add the table to the list of tables to be bootstrapped. > ***** All the subsequent events will be ignored as the new name is not m= atching the new policy. > #### New name matching new policy > ***** As the new name is not matching old policy but matching new policy= , the table will be bootstrapped by replace policy handler. So rename event= need not add this table to list of table to be bootstrapped. > ***** All the future events will be ignored by replace policy handler. > ***** For rename event (double rename) > ****** If there is a rename, the table (with intermittent new name) will= not be present and thus replace handler will not bootstrap the table. > ****** So if the new name (the latest one) is matching the new policy, t= hen add it to the list of table to be bootstrapped. > ****** And If the new name (the latest one)=C2=A0 is not matching the ne= w policy, then just ignore the event as the=C2=A0 intermittent new name wou= ld not have added to the list of table to be bootstrapped. > ### New name matching old policy > #### New name not matching new policy > ***** Dump the event. The table will be dropped by repl load at the targ= et. > #### New name matching new policy > ***** Replace handler will not bootstrap this table as the new name is m= atching both policies. > ***** As old name is not matching the old policy, the table will not be = there at target. The rename event should add the new name to the list of ta= ble to be bootstrapped. > ***** Subsequent events with new table name should be ignored. > ***** Drop events should not be ignored as if the table is present durin= g bootstrapped, then its a new table and thus should be dropped. > ***** In case of rename event (double rename) > ****** If the new name satisfies the table pattern, then add the new nam= e to the list of tables to be bootstrapped and remove the old name from the= list of tables to be bootstrapped. > ****** If the new name does not satisfies then just removed the table na= me from the list of tables to be bootstrapped. > # Old name is matching old policy =E2=80=93 The old table will be there = at the target cluster. The table will not be returned by get-all-table. Rep= l load should delete the old table as it is not matching the new policy. > ## Old name is not matching new policy > ### New name not matching old policy > #### New name not matching new policy > ***** Nothing to be done. Ignore the event. > #### New name matching new policy > ***** As the new name is not matching old policy but matching new policy= , the table will be bootstrapped by replace policy handler. So rename event= need not add this table to list of table to be bootstrapped. > ***** All the future events will be ignored by replace policy handler. > ***** For rename event (double rename) > ****** If there is a rename, the table (with intermittent new name) will= not be present and thus replace handler will not bootstrap the table. > ****** So if the new name (the latest one) is matching the new policy, t= hen add it to the list of table to be bootstrapped. > ****** And If the new name (the latest one)=C2=A0 is not matching the ne= w policy, then just ignore the event as the=C2=A0 intermittent new name wou= ld not have added to the list of table to be bootstrapped. > ### New name matching old policy > #### New name not matching new policy > ***** Table with new name will be dropped by repl load > ***** Along with other event, ignore the rename event also. > #### New name matching new policy > ***** As the new name is matching both old and new policy, replace handl= er will not bootstrap the table. > ***** Add the table to the list of tables to be bootstrapped. > ***** Ignore all the events with new name. > ***** If there is a drop event for the table (with new name), then remov= e the table from the the list of table to be bootstrapped. > ***** In case of rename event (double rename) > ****** If the new name satisfies the table pattern, then add the new nam= e to the list of tables to be bootstrapped and remove the old name from the= list of tables to be bootstrapped. > ****** If the new name does not satisfies then just removed the table na= me from the list of tables to be bootstrapped. > ## Old name is matching new policy > ### New name not matching old policy > #### New name not matching new policy > ***** The old table needs to be dropped at target. Ignore this event, as= the old table is not matching the new policy, it will be dropped by repl l= oad. > #### New name matching new policy > ***** Allow the event to dump and replayed at target. > ***** Allow further events to be handled as usual. > ***** In case of rename event (double rename) > ****** If the latest new name is matching the new policy, then keep it a= s is it. Let rename event replayed at target. > ****** If the latest new name is not matching the new policy, then chang= e the rename event to drop event. > ### New name matching old policy > #### New name not matching new policy > ##### Nothing to be done. > #### New name matching new policy > ##### Add the table name to the list of tables to be bootstrapped. -- This message was sent by Atlassian JIRA (v7.6.3#76005)