From issues-return-154919-archive-asf-public=cust-asf.ponee.io@hive.apache.org Mon Apr 1 15:34:04 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id DDA9E180778 for ; Mon, 1 Apr 2019 17:34:03 +0200 (CEST) Received: (qmail 4239 invoked by uid 500); 1 Apr 2019 15:34:02 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 4137 invoked by uid 99); 1 Apr 2019 15:34:02 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Apr 2019 15:34:02 +0000 Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 00EA9E2757 for ; Mon, 1 Apr 2019 15:34:02 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id C39A4245A4 for ; Mon, 1 Apr 2019 15:34:00 +0000 (UTC) Date: Mon, 1 Apr 2019 15:34:00 +0000 (UTC) From: "ASF GitHub Bot (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Work logged] (HIVE-21529) Hive support bootstrap of ACID/MM tables on an existing policy. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-21529?focusedWorklogId=3D= 221331&page=3Dcom.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpa= nel#worklog-221331 ] ASF GitHub Bot logged work on HIVE-21529: ----------------------------------------- Author: ASF GitHub Bot Created on: 01/Apr/19 15:33 Start Date: 01/Apr/19 15:33 Worklog Time Spent: 10m=20 Work Description: sankarh commented on pull request #581: HIVE-21529 = : Bootstrap ACID tables as part of incremental dump. URL: https://github.com/apache/hive/pull/581#discussion_r270896979 =20 =20 ########## File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.ja= va ########## @@ -193,27 +226,39 @@ private Long incrementalDump(Path dumpRoot, DumpMeta= Data dmd, Path cmRoot, Hive dmd.setDump(DumpType.INCREMENTAL, work.eventFrom, lastReplId, cmRoot); dmd.write(); =20 - // If external tables are enabled for replication and - // - If bootstrap is enabled, then need to combine bootstrap dump of e= xternal tables. - // - If metadata-only dump is enabled, then shall skip dumping externa= l tables data locations to - // _external_tables_info file. If not metadata-only, then dump the d= ata locations. - if (conf.getBoolVar(HiveConf.ConfVars.REPL_INCLUDE_EXTERNAL_TABLES) - && (!conf.getBoolVar(HiveConf.ConfVars.REPL_DUMP_METADATA_ONLY) - || conf.getBoolVar(HiveConf.ConfVars.REPL_BOOTSTRAP_EXTERNAL_TABLE= S))) { + // Examine all the tables if required. + if (shouldExamineTablesToDump()) { Path dbRoot =3D getBootstrapDbRoot(dumpRoot, dbName, true); + + // If we are bootstrapping ACID tables, stop all the concurrent tran= sactions and take a + // snapshot to dump those tables. Record the last event id in case w= e are performing + // bootstrap of ACID tables. + String validTxnList =3D null; + long bootstrapLastReplId =3D 0; + if (conf.getBoolVar(HiveConf.ConfVars.REPL_BOOTSTRAP_ACID_TABLES)) { + validTxnList =3D getValidTxnListForReplDump(hiveDb); + bootstrapLastReplId =3D hiveDb.getMSC().getCurrentNotificationEven= tId().getEventId(); =20 Review comment: bootstrapLastReplId should be captured before open txn of REPL DUMP quer= y.=20 =20 ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. =20 For queries about this service, please contact Infrastructure at: users@infra.apache.org Issue Time Tracking ------------------- Worklog Id: (was: 221331) Time Spent: 20m (was: 10m) > Hive support bootstrap of ACID/MM tables on an existing policy. > --------------------------------------------------------------- > > Key: HIVE-21529 > URL: https://issues.apache.org/jira/browse/HIVE-21529 > Project: Hive > Issue Type: Sub-task > Components: repl, Transactions > Affects Versions: 4.0.0 > Reporter: Sankar Hariappan > Assignee: Ashutosh Bapat > Priority: Major > Labels: DR, pull-request-available, replication > Attachments: HIVE-21529.01.patch > > Time Spent: 20m > Remaining Estimate: 0h > > If ACID/MM tables to be enabled (hive.repl.dump.include.acid.tables) on a= n existing repl policy, then need to combine bootstrap dump of these tables= along with the ongoing incremental dump.=20 > Shall add a one time config "hive.repl.bootstrap.acid.tables" to include= bootstrap in the given dump. > The=C2=A0support for hive.repl.bootstrap.cleanup.type for ACID tables to = clean-up partially bootstrapped tables in case of retry is already in place= , thanks to the work done during external tables. Need to test that it actu= ally works. -- This message was sent by Atlassian JIRA (v7.6.3#76005)